Jul 07 14:34:18.483019 ip-10-0-136-19 systemd[1]: Starting Kubernetes Kubelet... Jul 07 14:34:18.786518 ip-10-0-136-19 kubenswrapper[2555]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 14:34:18.786518 ip-10-0-136-19 kubenswrapper[2555]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 07 14:34:18.786518 ip-10-0-136-19 kubenswrapper[2555]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 14:34:18.786518 ip-10-0-136-19 kubenswrapper[2555]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 07 14:34:18.786518 ip-10-0-136-19 kubenswrapper[2555]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 14:34:18.787366 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.787264 2555 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 07 14:34:18.792373 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792358 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 14:34:18.792373 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792372 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 14:34:18.792440 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792376 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 14:34:18.792440 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792379 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 14:34:18.792440 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792383 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 14:34:18.792440 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792386 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 14:34:18.792440 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792390 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 14:34:18.792440 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792393 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 14:34:18.792440 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792396 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 14:34:18.792440 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792399 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 14:34:18.792440 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792403 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 14:34:18.792440 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792405 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 14:34:18.792440 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792408 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 14:34:18.792440 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792411 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 14:34:18.792440 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792413 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 14:34:18.792440 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792416 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 14:34:18.792440 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792419 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 14:34:18.792440 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792421 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 14:34:18.792440 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792424 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 14:34:18.792440 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792426 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 14:34:18.792440 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792429 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 14:34:18.792440 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792432 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 14:34:18.792905 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792434 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 14:34:18.792905 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792437 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 14:34:18.792905 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792439 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 14:34:18.792905 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792442 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 14:34:18.792905 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792445 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 14:34:18.792905 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792448 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 14:34:18.792905 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792450 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 14:34:18.792905 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792453 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 14:34:18.792905 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792457 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 14:34:18.792905 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792459 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 14:34:18.792905 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792462 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 14:34:18.792905 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792465 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 14:34:18.792905 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792468 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 14:34:18.792905 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792470 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 14:34:18.792905 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792473 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 14:34:18.792905 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792475 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 14:34:18.792905 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792478 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 14:34:18.792905 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792480 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 14:34:18.792905 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792483 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 14:34:18.792905 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792485 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 14:34:18.793388 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792488 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 14:34:18.793388 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792490 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 14:34:18.793388 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792493 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 14:34:18.793388 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792496 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 14:34:18.793388 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792498 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 14:34:18.793388 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792501 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 14:34:18.793388 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792503 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 14:34:18.793388 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792505 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 14:34:18.793388 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792508 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 14:34:18.793388 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792510 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 14:34:18.793388 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792513 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 14:34:18.793388 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792515 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 14:34:18.793388 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792518 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 14:34:18.793388 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792522 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 14:34:18.793388 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792525 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 14:34:18.793388 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792527 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 14:34:18.793388 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792530 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 14:34:18.793388 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792532 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 14:34:18.793388 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792535 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 14:34:18.793388 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792538 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 14:34:18.793883 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792540 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 14:34:18.793883 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792542 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 14:34:18.793883 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792545 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 14:34:18.793883 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792547 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 07 14:34:18.793883 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792553 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 14:34:18.793883 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792557 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 14:34:18.793883 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792560 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 14:34:18.793883 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792562 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 14:34:18.793883 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792565 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 14:34:18.793883 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792569 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 14:34:18.793883 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792573 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 14:34:18.793883 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792576 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 14:34:18.793883 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792579 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 14:34:18.793883 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792581 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 14:34:18.793883 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792584 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 14:34:18.793883 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792587 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 14:34:18.793883 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792592 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 14:34:18.793883 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792594 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 14:34:18.793883 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792597 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 14:34:18.794326 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792600 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 14:34:18.794326 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792603 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 14:34:18.794326 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792605 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 14:34:18.794326 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792608 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 14:34:18.794326 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792611 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 14:34:18.794326 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792615 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 14:34:18.794326 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792618 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 14:34:18.794326 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792621 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 14:34:18.794326 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792624 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 14:34:18.794326 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792627 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 14:34:18.794326 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792630 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 14:34:18.794326 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792632 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 14:34:18.794326 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792635 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 14:34:18.794326 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792639 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 14:34:18.794326 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792642 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 14:34:18.794326 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792645 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 14:34:18.794326 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792648 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 14:34:18.794326 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.792651 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 14:34:18.794326 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793391 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 14:34:18.794326 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793398 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 14:34:18.794814 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793401 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 14:34:18.794814 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793405 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 14:34:18.794814 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793409 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 14:34:18.794814 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793412 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 14:34:18.794814 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793415 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 14:34:18.794814 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793418 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 14:34:18.794814 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793421 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 14:34:18.794814 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793424 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 14:34:18.794814 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793426 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 14:34:18.794814 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793429 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 14:34:18.794814 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793432 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 14:34:18.794814 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793434 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 14:34:18.794814 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793437 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 14:34:18.794814 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793439 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 14:34:18.794814 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793442 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 14:34:18.794814 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793444 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 14:34:18.794814 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793447 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 14:34:18.794814 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793451 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 14:34:18.794814 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793455 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 14:34:18.794814 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793457 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 14:34:18.795288 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793460 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 14:34:18.795288 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793463 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 14:34:18.795288 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793465 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 14:34:18.795288 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793470 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 14:34:18.795288 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793473 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 14:34:18.795288 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793476 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 14:34:18.795288 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793478 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 14:34:18.795288 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793481 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 14:34:18.795288 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793483 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 14:34:18.795288 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793486 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 14:34:18.795288 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793489 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 14:34:18.795288 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793493 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 14:34:18.795288 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793496 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 14:34:18.795288 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793498 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 14:34:18.795288 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793502 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 14:34:18.795288 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793505 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 14:34:18.795288 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793509 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 14:34:18.795288 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793511 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 14:34:18.795288 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793514 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 14:34:18.795288 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793517 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 14:34:18.795776 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793519 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 14:34:18.795776 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793522 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 14:34:18.795776 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793527 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 14:34:18.795776 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793530 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 14:34:18.795776 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793534 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 14:34:18.795776 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793538 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 14:34:18.795776 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793540 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 14:34:18.795776 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793543 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 14:34:18.795776 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793545 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 14:34:18.795776 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793548 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 14:34:18.795776 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793551 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 14:34:18.795776 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793554 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 14:34:18.795776 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793557 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 14:34:18.795776 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793559 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 14:34:18.795776 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793562 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 14:34:18.795776 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793564 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 14:34:18.795776 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793567 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 14:34:18.795776 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793571 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 14:34:18.795776 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793574 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 14:34:18.796242 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793577 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 14:34:18.796242 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793580 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 14:34:18.796242 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793583 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 14:34:18.796242 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793586 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 14:34:18.796242 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793589 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 14:34:18.796242 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793592 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 14:34:18.796242 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793595 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 14:34:18.796242 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793598 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 14:34:18.796242 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793600 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 14:34:18.796242 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793603 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 14:34:18.796242 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793606 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 14:34:18.796242 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793609 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 14:34:18.796242 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793612 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 14:34:18.796242 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793614 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 14:34:18.796242 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793617 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 14:34:18.796242 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793620 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 14:34:18.796242 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793622 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 14:34:18.796242 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793625 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 14:34:18.796242 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793627 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 14:34:18.796242 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793629 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 14:34:18.796707 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793632 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 14:34:18.796707 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793634 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 14:34:18.796707 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793637 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 14:34:18.796707 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793640 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 14:34:18.796707 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793642 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 14:34:18.796707 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793645 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 14:34:18.796707 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793648 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 14:34:18.796707 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793651 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 14:34:18.796707 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793653 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 07 14:34:18.796707 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793656 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 14:34:18.796707 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793658 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 14:34:18.796707 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793661 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 14:34:18.796707 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793663 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 14:34:18.796707 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793666 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 14:34:18.796707 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793668 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 14:34:18.796707 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793671 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 14:34:18.796707 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793673 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 14:34:18.796707 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.793676 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 14:34:18.796707 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794081 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 14:34:18.797190 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794087 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 14:34:18.797190 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794091 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 14:34:18.797190 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794094 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 14:34:18.797190 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794099 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 14:34:18.797190 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794102 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 14:34:18.797190 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794105 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 14:34:18.797190 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794108 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 14:34:18.797190 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794111 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 14:34:18.797190 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794113 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 14:34:18.797190 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794116 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 14:34:18.797190 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794119 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 14:34:18.797190 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794121 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 14:34:18.797190 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794124 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 14:34:18.797190 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794128 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 14:34:18.797190 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794132 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 14:34:18.797190 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794135 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 14:34:18.797190 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794138 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 14:34:18.797190 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794140 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 14:34:18.797190 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794143 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 14:34:18.797677 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794146 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 14:34:18.797677 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794148 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 14:34:18.797677 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794151 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 14:34:18.797677 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794153 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 14:34:18.797677 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794156 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 14:34:18.797677 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794159 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 14:34:18.797677 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794161 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 14:34:18.797677 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794163 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 14:34:18.797677 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794166 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 14:34:18.797677 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794169 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 14:34:18.797677 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794171 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 14:34:18.797677 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794174 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 14:34:18.797677 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794177 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 14:34:18.797677 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794180 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 14:34:18.797677 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794183 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 14:34:18.797677 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794185 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 14:34:18.797677 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794188 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 14:34:18.797677 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794190 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 14:34:18.797677 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794193 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 14:34:18.797677 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794195 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 14:34:18.798160 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794198 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 14:34:18.798160 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794200 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 14:34:18.798160 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794203 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 14:34:18.798160 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794206 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 14:34:18.798160 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794208 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 14:34:18.798160 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794210 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 14:34:18.798160 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794213 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 14:34:18.798160 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794218 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 14:34:18.798160 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794221 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 14:34:18.798160 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794223 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 14:34:18.798160 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794226 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 14:34:18.798160 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794228 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 14:34:18.798160 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794231 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 14:34:18.798160 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794233 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 14:34:18.798160 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794236 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 14:34:18.798160 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794238 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 14:34:18.798160 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794241 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 14:34:18.798160 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794244 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 14:34:18.798160 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794246 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 14:34:18.798160 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794249 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 14:34:18.798628 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794251 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 14:34:18.798628 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794254 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 14:34:18.798628 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794256 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 14:34:18.798628 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794259 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 14:34:18.798628 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794261 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 14:34:18.798628 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794265 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 14:34:18.798628 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794267 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 14:34:18.798628 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794270 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 14:34:18.798628 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794272 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 14:34:18.798628 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794275 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 14:34:18.798628 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794277 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 14:34:18.798628 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794280 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 14:34:18.798628 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794282 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 14:34:18.798628 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794285 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 14:34:18.798628 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794287 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 14:34:18.798628 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794290 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 14:34:18.798628 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794292 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 07 14:34:18.798628 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794295 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 14:34:18.798628 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794298 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 14:34:18.799186 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794302 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 14:34:18.799186 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794306 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 14:34:18.799186 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794309 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 14:34:18.799186 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794312 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 14:34:18.799186 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794315 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 14:34:18.799186 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794318 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 14:34:18.799186 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794320 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 14:34:18.799186 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794323 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 14:34:18.799186 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794325 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 14:34:18.799186 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794328 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 14:34:18.799186 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794330 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 14:34:18.799186 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794333 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 14:34:18.799186 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794336 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 14:34:18.799186 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794339 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 14:34:18.799186 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794341 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 14:34:18.799186 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794344 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 14:34:18.799186 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794346 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 14:34:18.799186 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794350 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 14:34:18.799186 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794353 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 14:34:18.799186 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.794356 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 14:34:18.799677 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794417 2555 flags.go:64] FLAG: --address="0.0.0.0" Jul 07 14:34:18.799677 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794423 2555 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 07 14:34:18.799677 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794430 2555 flags.go:64] FLAG: --anonymous-auth="true" Jul 07 14:34:18.799677 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794434 2555 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 07 14:34:18.799677 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794438 2555 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 07 14:34:18.799677 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794441 2555 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 07 14:34:18.799677 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794445 2555 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 07 14:34:18.799677 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794449 2555 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 07 14:34:18.799677 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794452 2555 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 07 14:34:18.799677 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794455 2555 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 07 14:34:18.799677 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794458 2555 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 07 14:34:18.799677 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794461 2555 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 07 14:34:18.799677 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794466 2555 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 07 14:34:18.799677 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794469 2555 flags.go:64] FLAG: --cgroup-root="" Jul 07 14:34:18.799677 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794472 2555 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 07 14:34:18.799677 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794475 2555 flags.go:64] FLAG: --client-ca-file="" Jul 07 14:34:18.799677 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794478 2555 flags.go:64] FLAG: --cloud-provider="external" Jul 07 14:34:18.799677 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794481 2555 flags.go:64] FLAG: --cluster-dns="[]" Jul 07 14:34:18.799677 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794485 2555 flags.go:64] FLAG: --cluster-domain="" Jul 07 14:34:18.799677 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794488 2555 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 07 14:34:18.799677 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794491 2555 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 07 14:34:18.799677 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794497 2555 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 07 14:34:18.799677 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794500 2555 flags.go:64] FLAG: --container-log-max-files="5" Jul 07 14:34:18.799677 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794504 2555 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 07 14:34:18.800268 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794507 2555 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 07 14:34:18.800268 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794510 2555 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 07 14:34:18.800268 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794513 2555 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 07 14:34:18.800268 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794516 2555 flags.go:64] FLAG: --contention-profiling="false" Jul 07 14:34:18.800268 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794519 2555 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 07 14:34:18.800268 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794522 2555 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 07 14:34:18.800268 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794525 2555 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 07 14:34:18.800268 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794528 2555 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 07 14:34:18.800268 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794535 2555 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 07 14:34:18.800268 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794539 2555 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 07 14:34:18.800268 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794542 2555 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 07 14:34:18.800268 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794544 2555 flags.go:64] FLAG: --enable-load-reader="false" Jul 07 14:34:18.800268 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794547 2555 flags.go:64] FLAG: --enable-server="true" Jul 07 14:34:18.800268 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794550 2555 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 07 14:34:18.800268 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794555 2555 flags.go:64] FLAG: --event-burst="100" Jul 07 14:34:18.800268 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794558 2555 flags.go:64] FLAG: --event-qps="50" Jul 07 14:34:18.800268 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794561 2555 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 07 14:34:18.800268 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794564 2555 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 07 14:34:18.800268 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794567 2555 flags.go:64] FLAG: --eviction-hard="" Jul 07 14:34:18.800268 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794570 2555 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 07 14:34:18.800268 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794575 2555 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 07 14:34:18.800268 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794577 2555 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 07 14:34:18.800268 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794580 2555 flags.go:64] FLAG: --eviction-soft="" Jul 07 14:34:18.800268 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794583 2555 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 07 14:34:18.800268 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794586 2555 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 07 14:34:18.800929 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794589 2555 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 07 14:34:18.800929 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794592 2555 flags.go:64] FLAG: --experimental-mounter-path="" Jul 07 14:34:18.800929 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794595 2555 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 07 14:34:18.800929 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794597 2555 flags.go:64] FLAG: --fail-swap-on="true" Jul 07 14:34:18.800929 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794602 2555 flags.go:64] FLAG: --feature-gates="" Jul 07 14:34:18.800929 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794605 2555 flags.go:64] FLAG: --file-check-frequency="20s" Jul 07 14:34:18.800929 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794608 2555 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 07 14:34:18.800929 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794611 2555 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 07 14:34:18.800929 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794614 2555 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 07 14:34:18.800929 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794617 2555 flags.go:64] FLAG: --healthz-port="10248" Jul 07 14:34:18.800929 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794620 2555 flags.go:64] FLAG: --help="false" Jul 07 14:34:18.800929 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794623 2555 flags.go:64] FLAG: --hostname-override="ip-10-0-136-19.ec2.internal" Jul 07 14:34:18.800929 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794626 2555 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 07 14:34:18.800929 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794629 2555 flags.go:64] FLAG: --http-check-frequency="20s" Jul 07 14:34:18.800929 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794632 2555 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 07 14:34:18.800929 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794635 2555 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 07 14:34:18.800929 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794639 2555 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 07 14:34:18.800929 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794642 2555 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 07 14:34:18.800929 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794645 2555 flags.go:64] FLAG: --image-service-endpoint="" Jul 07 14:34:18.800929 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794647 2555 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 07 14:34:18.800929 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794650 2555 flags.go:64] FLAG: --kube-api-burst="100" Jul 07 14:34:18.800929 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794653 2555 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 07 14:34:18.800929 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794656 2555 flags.go:64] FLAG: --kube-api-qps="50" Jul 07 14:34:18.800929 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794658 2555 flags.go:64] FLAG: --kube-reserved="" Jul 07 14:34:18.801549 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794661 2555 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 07 14:34:18.801549 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794664 2555 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 07 14:34:18.801549 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794667 2555 flags.go:64] FLAG: --kubelet-cgroups="" Jul 07 14:34:18.801549 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794672 2555 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 07 14:34:18.801549 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794675 2555 flags.go:64] FLAG: --lock-file="" Jul 07 14:34:18.801549 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794677 2555 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 07 14:34:18.801549 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794680 2555 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 07 14:34:18.801549 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794683 2555 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 07 14:34:18.801549 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794689 2555 flags.go:64] FLAG: --log-json-split-stream="false" Jul 07 14:34:18.801549 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794691 2555 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 07 14:34:18.801549 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794694 2555 flags.go:64] FLAG: --log-text-split-stream="false" Jul 07 14:34:18.801549 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794697 2555 flags.go:64] FLAG: --logging-format="text" Jul 07 14:34:18.801549 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794701 2555 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 07 14:34:18.801549 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794705 2555 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 07 14:34:18.801549 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794708 2555 flags.go:64] FLAG: --manifest-url="" Jul 07 14:34:18.801549 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794710 2555 flags.go:64] FLAG: --manifest-url-header="" Jul 07 14:34:18.801549 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794714 2555 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 07 14:34:18.801549 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794718 2555 flags.go:64] FLAG: --max-open-files="1000000" Jul 07 14:34:18.801549 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794722 2555 flags.go:64] FLAG: --max-pods="110" Jul 07 14:34:18.801549 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794725 2555 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 07 14:34:18.801549 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794728 2555 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 07 14:34:18.801549 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794730 2555 flags.go:64] FLAG: --memory-manager-policy="None" Jul 07 14:34:18.801549 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794733 2555 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 07 14:34:18.801549 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794736 2555 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 07 14:34:18.801549 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794739 2555 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 07 14:34:18.802163 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794742 2555 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 07 14:34:18.802163 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794764 2555 flags.go:64] FLAG: --node-status-max-images="50" Jul 07 14:34:18.802163 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794769 2555 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 07 14:34:18.802163 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794773 2555 flags.go:64] FLAG: --oom-score-adj="-999" Jul 07 14:34:18.802163 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794776 2555 flags.go:64] FLAG: --pod-cidr="" Jul 07 14:34:18.802163 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794778 2555 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 07 14:34:18.802163 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794784 2555 flags.go:64] FLAG: --pod-manifest-path="" Jul 07 14:34:18.802163 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794787 2555 flags.go:64] FLAG: --pod-max-pids="-1" Jul 07 14:34:18.802163 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794790 2555 flags.go:64] FLAG: --pods-per-core="0" Jul 07 14:34:18.802163 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794793 2555 flags.go:64] FLAG: --port="10250" Jul 07 14:34:18.802163 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794796 2555 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 07 14:34:18.802163 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794800 2555 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b344bb68f80f102b" Jul 07 14:34:18.802163 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794804 2555 flags.go:64] FLAG: --qos-reserved="" Jul 07 14:34:18.802163 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794807 2555 flags.go:64] FLAG: --read-only-port="10255" Jul 07 14:34:18.802163 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794810 2555 flags.go:64] FLAG: --register-node="true" Jul 07 14:34:18.802163 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794813 2555 flags.go:64] FLAG: --register-with-taints="" Jul 07 14:34:18.802163 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794816 2555 flags.go:64] FLAG: --registry-burst="10" Jul 07 14:34:18.802163 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794819 2555 flags.go:64] FLAG: --registry-qps="5" Jul 07 14:34:18.802163 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794822 2555 flags.go:64] FLAG: --reserved-cpus="" Jul 07 14:34:18.802163 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794826 2555 flags.go:64] FLAG: --reserved-memory="" Jul 07 14:34:18.802163 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794830 2555 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 07 14:34:18.802163 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794833 2555 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 07 14:34:18.802163 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794836 2555 flags.go:64] FLAG: --rotate-certificates="false" Jul 07 14:34:18.802163 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794839 2555 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 07 14:34:18.802163 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794841 2555 flags.go:64] FLAG: --runonce="false" Jul 07 14:34:18.803189 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794844 2555 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 07 14:34:18.803189 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794848 2555 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 07 14:34:18.803189 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794850 2555 flags.go:64] FLAG: --seccomp-default="false" Jul 07 14:34:18.803189 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794853 2555 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 07 14:34:18.803189 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794856 2555 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 07 14:34:18.803189 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794859 2555 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 07 14:34:18.803189 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794862 2555 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 07 14:34:18.803189 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794865 2555 flags.go:64] FLAG: --storage-driver-password="root" Jul 07 14:34:18.803189 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794868 2555 flags.go:64] FLAG: --storage-driver-secure="false" Jul 07 14:34:18.803189 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794871 2555 flags.go:64] FLAG: --storage-driver-table="stats" Jul 07 14:34:18.803189 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794874 2555 flags.go:64] FLAG: --storage-driver-user="root" Jul 07 14:34:18.803189 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794877 2555 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 07 14:34:18.803189 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794880 2555 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 07 14:34:18.803189 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794883 2555 flags.go:64] FLAG: --system-cgroups="" Jul 07 14:34:18.803189 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794885 2555 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 07 14:34:18.803189 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794891 2555 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 07 14:34:18.803189 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794894 2555 flags.go:64] FLAG: --tls-cert-file="" Jul 07 14:34:18.803189 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794897 2555 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 07 14:34:18.803189 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794902 2555 flags.go:64] FLAG: --tls-min-version="" Jul 07 14:34:18.803189 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794905 2555 flags.go:64] FLAG: --tls-private-key-file="" Jul 07 14:34:18.803189 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794908 2555 flags.go:64] FLAG: --topology-manager-policy="none" Jul 07 14:34:18.803189 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794911 2555 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 07 14:34:18.803189 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794914 2555 flags.go:64] FLAG: --topology-manager-scope="container" Jul 07 14:34:18.803189 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794916 2555 flags.go:64] FLAG: --v="2" Jul 07 14:34:18.803189 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794920 2555 flags.go:64] FLAG: --version="false" Jul 07 14:34:18.803893 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794924 2555 flags.go:64] FLAG: --vmodule="" Jul 07 14:34:18.803893 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794929 2555 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 07 14:34:18.803893 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.794932 2555 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 07 14:34:18.803893 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795045 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 14:34:18.803893 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795049 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 14:34:18.803893 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795052 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 14:34:18.803893 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795055 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 14:34:18.803893 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795059 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 14:34:18.803893 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795063 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 14:34:18.803893 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795066 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 14:34:18.803893 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795068 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 14:34:18.803893 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795071 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 14:34:18.803893 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795073 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 14:34:18.803893 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795076 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 14:34:18.803893 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795078 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 14:34:18.803893 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795081 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 14:34:18.803893 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795084 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 14:34:18.803893 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795087 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 14:34:18.803893 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795089 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 14:34:18.803893 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795092 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 14:34:18.804502 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795094 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 14:34:18.804502 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795097 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 14:34:18.804502 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795100 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 14:34:18.804502 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795103 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 14:34:18.804502 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795105 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 14:34:18.804502 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795109 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 14:34:18.804502 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795112 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 14:34:18.804502 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795115 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 14:34:18.804502 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795118 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 14:34:18.804502 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795120 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 14:34:18.804502 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795123 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 14:34:18.804502 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795125 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 14:34:18.804502 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795128 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 14:34:18.804502 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795131 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 14:34:18.804502 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795134 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 14:34:18.804502 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795136 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 14:34:18.804502 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795139 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 14:34:18.804502 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795142 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 14:34:18.804502 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795144 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 14:34:18.804502 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795147 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 14:34:18.805017 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795149 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 14:34:18.805017 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795152 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 14:34:18.805017 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795155 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 14:34:18.805017 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795159 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 14:34:18.805017 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795162 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 14:34:18.805017 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795165 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 14:34:18.805017 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795168 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 14:34:18.805017 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795171 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 14:34:18.805017 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795174 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 14:34:18.805017 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795177 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 14:34:18.805017 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795180 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 14:34:18.805017 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795182 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 14:34:18.805017 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795185 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 14:34:18.805017 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795187 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 14:34:18.805017 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795190 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 14:34:18.805017 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795192 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 14:34:18.805017 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795195 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 14:34:18.805017 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795198 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 14:34:18.805017 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795201 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 14:34:18.805017 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795203 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 14:34:18.805505 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795206 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 14:34:18.805505 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795209 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 14:34:18.805505 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795211 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 14:34:18.805505 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795214 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 14:34:18.805505 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795217 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 14:34:18.805505 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795220 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 14:34:18.805505 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795223 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 14:34:18.805505 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795225 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 14:34:18.805505 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795228 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 14:34:18.805505 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795230 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 14:34:18.805505 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795233 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 14:34:18.805505 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795236 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 14:34:18.805505 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795238 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 14:34:18.805505 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795241 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 14:34:18.805505 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795243 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 14:34:18.805505 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795246 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 14:34:18.805505 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795248 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 14:34:18.805505 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795251 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 14:34:18.805505 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795253 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 14:34:18.805974 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795256 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 14:34:18.805974 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795258 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 14:34:18.805974 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795260 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 14:34:18.805974 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795263 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 14:34:18.805974 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795266 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 14:34:18.805974 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795268 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 14:34:18.805974 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795271 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 14:34:18.805974 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795274 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 14:34:18.805974 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795277 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 14:34:18.805974 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795279 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 14:34:18.805974 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795283 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 14:34:18.805974 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795286 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 14:34:18.805974 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795288 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 14:34:18.805974 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795291 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 14:34:18.805974 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795294 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 14:34:18.805974 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795296 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 14:34:18.805974 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795299 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 14:34:18.805974 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795302 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 14:34:18.805974 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795306 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 14:34:18.806428 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795310 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 14:34:18.806428 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795313 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 14:34:18.806428 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795316 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 14:34:18.806428 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.795319 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 07 14:34:18.806428 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.795821 2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 14:34:18.806428 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.804793 2555 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 07 14:34:18.806428 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.804809 2555 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 07 14:34:18.806428 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.805390 2555 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 07 14:34:18.806428 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.805398 2555 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 07 14:34:18.806428 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805455 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 14:34:18.806428 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805460 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 14:34:18.806428 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805464 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 14:34:18.806428 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805467 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 14:34:18.806428 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805470 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 14:34:18.806428 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805473 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 14:34:18.806851 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805476 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 14:34:18.806851 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805478 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 14:34:18.806851 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805481 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 14:34:18.806851 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805484 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 14:34:18.806851 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805486 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 14:34:18.806851 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805489 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 14:34:18.806851 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805492 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 14:34:18.806851 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805495 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 14:34:18.806851 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805497 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 14:34:18.806851 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805500 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 14:34:18.806851 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805503 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 14:34:18.806851 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805505 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 14:34:18.806851 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805508 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 14:34:18.806851 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805511 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 14:34:18.806851 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805514 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 14:34:18.806851 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805517 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 14:34:18.806851 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805520 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 14:34:18.806851 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805523 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 14:34:18.806851 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805525 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 14:34:18.806851 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805528 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 14:34:18.807337 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805531 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 14:34:18.807337 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805534 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 14:34:18.807337 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805537 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 07 14:34:18.807337 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805540 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 14:34:18.807337 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805543 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 14:34:18.807337 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805546 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 14:34:18.807337 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805549 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 14:34:18.807337 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805551 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 14:34:18.807337 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805554 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 14:34:18.807337 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805556 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 14:34:18.807337 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805559 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 14:34:18.807337 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805561 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 14:34:18.807337 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805564 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 14:34:18.807337 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805567 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 14:34:18.807337 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805569 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 14:34:18.807337 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805572 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 14:34:18.807337 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805574 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 14:34:18.807337 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805577 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 14:34:18.807337 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805579 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 14:34:18.807825 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805582 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 14:34:18.807825 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805585 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 14:34:18.807825 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805588 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 14:34:18.807825 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805590 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 14:34:18.807825 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805593 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 14:34:18.807825 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805595 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 14:34:18.807825 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805598 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 14:34:18.807825 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805600 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 14:34:18.807825 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805604 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 14:34:18.807825 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805608 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 14:34:18.807825 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805611 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 14:34:18.807825 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805614 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 14:34:18.807825 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805617 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 14:34:18.807825 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805619 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 14:34:18.807825 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805622 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 14:34:18.807825 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805624 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 14:34:18.807825 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805628 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 14:34:18.807825 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805631 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 14:34:18.807825 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805634 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 14:34:18.807825 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805636 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 14:34:18.808326 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805639 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 14:34:18.808326 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805642 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 14:34:18.808326 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805644 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 14:34:18.808326 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805647 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 14:34:18.808326 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805650 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 14:34:18.808326 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805652 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 14:34:18.808326 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805655 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 14:34:18.808326 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805657 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 14:34:18.808326 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805660 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 14:34:18.808326 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805662 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 14:34:18.808326 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805666 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 14:34:18.808326 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805670 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 14:34:18.808326 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805673 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 14:34:18.808326 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805676 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 14:34:18.808326 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805679 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 14:34:18.808326 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805682 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 14:34:18.808326 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805685 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 14:34:18.808326 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805688 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 14:34:18.808326 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805691 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 14:34:18.808794 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805695 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 14:34:18.808794 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805698 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 14:34:18.808794 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805700 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 14:34:18.808794 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805703 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 14:34:18.808794 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805706 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 14:34:18.808794 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805708 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 14:34:18.808794 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805711 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 14:34:18.808794 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805714 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 14:34:18.808794 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805717 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 14:34:18.808794 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805720 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 14:34:18.808794 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805722 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 14:34:18.808794 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805725 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 14:34:18.808794 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805727 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 14:34:18.808794 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805730 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 14:34:18.808794 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805733 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 14:34:18.809189 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.805738 2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 14:34:18.809189 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805865 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 14:34:18.809189 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805870 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 14:34:18.809189 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805873 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 14:34:18.809189 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805877 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 14:34:18.809189 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805880 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 14:34:18.809189 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805882 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 14:34:18.809189 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805885 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 14:34:18.809189 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805888 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 14:34:18.809189 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805890 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 14:34:18.809189 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805893 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 14:34:18.809189 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805895 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 14:34:18.809189 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805898 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 14:34:18.809189 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805901 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 14:34:18.809189 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805904 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 14:34:18.809570 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805907 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 14:34:18.809570 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805909 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 14:34:18.809570 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805912 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 14:34:18.809570 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805914 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 14:34:18.809570 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805917 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 14:34:18.809570 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805919 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 14:34:18.809570 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805922 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 14:34:18.809570 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805924 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 14:34:18.809570 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805927 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 14:34:18.809570 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805929 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 14:34:18.809570 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805932 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 14:34:18.809570 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805936 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 14:34:18.809570 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805940 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 14:34:18.809570 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805943 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 14:34:18.809570 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805946 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 14:34:18.809570 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805949 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 14:34:18.809570 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805952 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 14:34:18.809570 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805954 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 14:34:18.809570 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805957 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 14:34:18.810066 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805960 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 14:34:18.810066 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805963 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 14:34:18.810066 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805965 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 14:34:18.810066 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805968 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 14:34:18.810066 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805970 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 14:34:18.810066 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805973 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 14:34:18.810066 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805976 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 14:34:18.810066 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805978 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 14:34:18.810066 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805981 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 14:34:18.810066 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805984 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 14:34:18.810066 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805986 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 14:34:18.810066 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805989 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 14:34:18.810066 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805993 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 14:34:18.810066 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805995 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 14:34:18.810066 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.805998 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 14:34:18.810066 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806000 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 14:34:18.810066 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806004 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 14:34:18.810066 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806007 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 14:34:18.810066 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806010 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 14:34:18.810533 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806013 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 14:34:18.810533 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806015 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 14:34:18.810533 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806018 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 14:34:18.810533 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806021 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 14:34:18.810533 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806024 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 14:34:18.810533 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806026 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 14:34:18.810533 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806029 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 14:34:18.810533 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806031 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 14:34:18.810533 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806034 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 14:34:18.810533 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806037 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 14:34:18.810533 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806039 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 14:34:18.810533 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806042 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 14:34:18.810533 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806044 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 14:34:18.810533 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806047 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 14:34:18.810533 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806049 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 07 14:34:18.810533 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806052 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 14:34:18.810533 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806054 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 14:34:18.810533 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806057 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 14:34:18.810533 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806059 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 14:34:18.810533 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806062 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 14:34:18.811028 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806064 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 14:34:18.811028 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806067 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 14:34:18.811028 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806069 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 14:34:18.811028 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806072 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 14:34:18.811028 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806075 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 14:34:18.811028 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806077 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 14:34:18.811028 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806080 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 14:34:18.811028 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806083 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 14:34:18.811028 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806085 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 14:34:18.811028 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806088 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 14:34:18.811028 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806090 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 14:34:18.811028 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806093 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 14:34:18.811028 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806095 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 14:34:18.811028 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806098 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 14:34:18.811028 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806101 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 14:34:18.811028 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806104 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 14:34:18.811028 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806108 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 14:34:18.811028 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806111 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 14:34:18.811028 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806114 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 14:34:18.811486 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806116 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 14:34:18.811486 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806119 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 14:34:18.811486 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806122 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 14:34:18.811486 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806124 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 14:34:18.811486 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806127 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 14:34:18.811486 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806130 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 14:34:18.811486 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806133 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 14:34:18.811486 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:18.806136 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 14:34:18.811486 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.806141 2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 14:34:18.811486 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.806257 2555 server.go:962] "Client rotation is on, will bootstrap in background" Jul 07 14:34:18.811486 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.809197 2555 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 07 14:34:18.811486 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.810262 2555 server.go:1019] "Starting client certificate rotation" Jul 07 14:34:18.811486 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.810366 2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 14:34:18.811486 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.810396 2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 14:34:18.812331 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.812318 2555 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 14:34:18.816087 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.816071 2555 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 14:34:18.827426 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.827406 2555 log.go:25] "Validated CRI v1 runtime API" Jul 07 14:34:18.829817 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.829804 2555 log.go:25] "Validated CRI v1 image API" Jul 07 14:34:18.831165 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.831152 2555 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 07 14:34:18.833218 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.833189 2555 fs.go:135] Filesystem UUIDs: map[0bbccf53-f9bb-430d-b11f-42c6cec62f58:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 96b84033-0b3b-49f3-a6de-93ec0c5a1378:/dev/nvme0n1p3] Jul 07 14:34:18.833292 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.833217 2555 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 07 14:34:18.838840 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.838616 2555 manager.go:216] Machine: {Timestamp:2026-07-07 14:34:18.836676054 +0000 UTC m=+0.260862521 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099836 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2402844cc8b5c150cadaa58e8b9ad2 SystemUUID:ec240284-4cc8-b5c1-50ca-daa58e8b9ad2 BootID:7109a8a9-712c-460e-a228-d4d7139c3e8b Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:13:50:ba:62:af Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:13:50:ba:62:af Speed:0 Mtu:9001} {Name:ovs-system MacAddress:32:ee:ba:79:ee:9f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 07 14:34:18.838840 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.838830 2555 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 07 14:34:18.839250 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.839228 2555 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 07 14:34:18.841333 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.841313 2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 14:34:18.841789 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.841745 2555 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 07 14:34:18.841918 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.841790 2555 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-19.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 07 14:34:18.842340 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.842330 2555 topology_manager.go:138] "Creating topology manager with none policy" Jul 07 14:34:18.842378 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.842343 2555 container_manager_linux.go:309] "Creating device plugin manager" Jul 07 14:34:18.842378 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.842350 2555 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 14:34:18.842946 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.842935 2555 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 14:34:18.844049 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.844038 2555 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 07 14:34:18.847933 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.847922 2555 state_mem.go:36] "Initialized new in-memory state store" Jul 07 14:34:18.848048 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.848039 2555 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 07 14:34:18.848079 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.848067 2555 kubelet.go:486] "Attempting to sync node with API server" Jul 07 14:34:18.848079 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.848076 2555 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 07 14:34:18.848130 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.848091 2555 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 07 14:34:18.848130 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.848101 2555 kubelet.go:398] "Adding apiserver pod source" Jul 07 14:34:18.848130 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.848108 2555 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 07 14:34:18.848502 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.848490 2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 14:34:18.848539 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.848509 2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 14:34:18.850954 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.850937 2555 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 07 14:34:18.852701 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.852685 2555 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 07 14:34:18.852804 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.852716 2555 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 07 14:34:18.853698 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.853686 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 07 14:34:18.853741 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.853703 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 07 14:34:18.853741 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.853709 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 07 14:34:18.853741 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.853715 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 07 14:34:18.853741 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.853723 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 07 14:34:18.853741 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.853729 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 07 14:34:18.853888 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.853743 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 07 14:34:18.853888 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.853765 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 07 14:34:18.853888 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.853773 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 07 14:34:18.853888 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.853778 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 07 14:34:18.853888 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.853787 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 07 14:34:18.853888 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.853793 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 07 14:34:18.855069 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.855056 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 07 14:34:18.855069 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.855068 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 07 14:34:18.856858 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:18.856833 2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 07 14:34:18.856912 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:18.856833 2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 07 14:34:18.856912 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.856896 2555 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 07 14:34:18.858320 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.858307 2555 server.go:1268] "Started kubelet" Jul 07 14:34:18.858499 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.858369 2555 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 07 14:34:18.858499 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.858406 2555 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 07 14:34:18.858499 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.858477 2555 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 07 14:34:18.859225 ip-10-0-136-19 systemd[1]: Started Kubernetes Kubelet. Jul 07 14:34:18.859630 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.859599 2555 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 07 14:34:18.860419 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.860405 2555 server.go:310] "Adding debug handlers to kubelet server" Jul 07 14:34:18.864509 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:18.864490 2555 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 07 14:34:18.864869 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.864853 2555 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 07 14:34:18.864927 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.864857 2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 07 14:34:18.865065 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:18.865041 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-19.ec2.internal\" not found" Jul 07 14:34:18.865405 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.865368 2555 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 07 14:34:18.865492 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.865424 2555 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 07 14:34:18.865492 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.865446 2555 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 07 14:34:18.865588 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.865527 2555 reconstruct.go:97] "Volume reconstruction finished" Jul 07 14:34:18.865588 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.865536 2555 reconciler.go:29] "Reconciler: start to sync state" Jul 07 14:34:18.865718 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.865703 2555 factory.go:55] Registering systemd factory Jul 07 14:34:18.865781 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.865722 2555 factory.go:223] Registration of the systemd container factory successfully Jul 07 14:34:18.865946 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.865932 2555 factory.go:153] Registering CRI-O factory Jul 07 14:34:18.865946 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.865948 2555 factory.go:223] Registration of the crio container factory successfully Jul 07 14:34:18.866077 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.865998 2555 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 07 14:34:18.866077 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.866019 2555 factory.go:103] Registering Raw factory Jul 07 14:34:18.866077 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.866033 2555 manager.go:1196] Started watching for new ooms in manager Jul 07 14:34:18.866483 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.866466 2555 manager.go:349] Starting recovery of all containers Jul 07 14:34:18.868558 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:18.868530 2555 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 07 14:34:18.871590 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:18.871562 2555 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 07 14:34:18.872150 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:18.868587 2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-19.ec2.internal.18c0088e1a4be408 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-19.ec2.internal,UID:ip-10-0-136-19.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-19.ec2.internal,},FirstTimestamp:2026-07-07 14:34:18.858275848 +0000 UTC m=+0.282462313,LastTimestamp:2026-07-07 14:34:18.858275848 +0000 UTC m=+0.282462313,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-19.ec2.internal,}" Jul 07 14:34:18.876393 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.876371 2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z78jp" Jul 07 14:34:18.880823 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.880806 2555 manager.go:354] Recovery completed Jul 07 14:34:18.884739 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.884719 2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z78jp" Jul 07 14:34:18.885118 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.885106 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 14:34:18.886825 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.886809 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-19.ec2.internal" event="NodeHasSufficientMemory" Jul 07 14:34:18.886888 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.886836 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-19.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 14:34:18.886888 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.886846 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-19.ec2.internal" event="NodeHasSufficientPID" Jul 07 14:34:18.887368 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.887355 2555 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 07 14:34:18.887427 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.887369 2555 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 07 14:34:18.887427 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.887387 2555 state_mem.go:36] "Initialized new in-memory state store" Jul 07 14:34:18.887427 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.887410 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 14:34:18.888079 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.888064 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-19.ec2.internal" event="NodeHasSufficientMemory" Jul 07 14:34:18.888137 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.888091 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-19.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 14:34:18.888137 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.888103 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-19.ec2.internal" event="NodeHasSufficientPID" Jul 07 14:34:18.888844 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:18.888773 2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-19.ec2.internal.18c0088e1bff7ce4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-19.ec2.internal,UID:ip-10-0-136-19.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-19.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-19.ec2.internal,},FirstTimestamp:2026-07-07 14:34:18.88682314 +0000 UTC m=+0.311009606,LastTimestamp:2026-07-07 14:34:18.88682314 +0000 UTC m=+0.311009606,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-19.ec2.internal,}" Jul 07 14:34:18.889310 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.889299 2555 policy_none.go:49] "None policy: Start" Jul 07 14:34:18.889347 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.889314 2555 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 07 14:34:18.889347 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.889324 2555 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 07 14:34:18.891365 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.891356 2555 policy_none.go:47] "Start" Jul 07 14:34:18.930401 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.930276 2555 manager.go:337] "Starting Device Plugin manager" Jul 07 14:34:18.930496 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:18.930414 2555 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 07 14:34:18.930496 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.930427 2555 server.go:87] "Starting device plugin registration server" Jul 07 14:34:18.930662 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.930649 2555 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 07 14:34:18.930710 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.930663 2555 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 07 14:34:18.930780 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.930729 2555 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 07 14:34:18.930839 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.930819 2555 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 07 14:34:18.930839 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:18.930829 2555 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 07 14:34:18.931256 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:18.931236 2555 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 07 14:34:18.931333 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:18.931282 2555 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-19.ec2.internal\" not found" Jul 07 14:34:19.031196 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:19.031166 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 14:34:19.033262 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:19.033147 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-19.ec2.internal" event="NodeHasSufficientMemory" Jul 07 14:34:19.033262 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:19.033182 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-19.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 14:34:19.033262 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:19.033196 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-19.ec2.internal" event="NodeHasSufficientPID" Jul 07 14:34:19.033262 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:19.033227 2555 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-19.ec2.internal" Jul 07 14:34:19.034306 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:19.034276 2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 07 14:34:19.035529 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:19.035506 2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 07 14:34:19.035659 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:19.035551 2555 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 07 14:34:19.035659 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:19.035581 2555 kubelet.go:2448] "Starting kubelet main sync loop" Jul 07 14:34:19.035659 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:19.035620 2555 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 07 14:34:19.038285 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:19.038239 2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 14:34:19.040240 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:19.040224 2555 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-19.ec2.internal" Jul 07 14:34:19.040306 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:19.040245 2555 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-19.ec2.internal\": node \"ip-10-0-136-19.ec2.internal\" not found" Jul 07 14:34:19.094888 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:19.094861 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-19.ec2.internal\" not found" Jul 07 14:34:19.136170 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:19.136146 2555 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-19.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-19.ec2.internal"] Jul 07 14:34:19.136296 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:19.136211 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 14:34:19.137177 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:19.137158 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-19.ec2.internal" event="NodeHasSufficientMemory" Jul 07 14:34:19.137280 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:19.137192 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-19.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 14:34:19.137280 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:19.137207 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-19.ec2.internal" event="NodeHasSufficientPID" Jul 07 14:34:19.139459 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:19.139444 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 14:34:19.139547 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:19.139521 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-19.ec2.internal" Jul 07 14:34:19.139547 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:19.139546 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 14:34:19.140196 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:19.140182 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-19.ec2.internal" event="NodeHasSufficientMemory" Jul 07 14:34:19.140271 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:19.140203 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-19.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 14:34:19.140271 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:19.140212 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-19.ec2.internal" event="NodeHasSufficientPID" Jul 07 14:34:19.140271 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:19.140251 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-19.ec2.internal" event="NodeHasSufficientMemory" Jul 07 14:34:19.140406 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:19.140273 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-19.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 14:34:19.140406 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:19.140286 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-19.ec2.internal" event="NodeHasSufficientPID" Jul 07 14:34:19.142445 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:19.142429 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-19.ec2.internal" Jul 07 14:34:19.142506 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:19.142456 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 14:34:19.143050 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:19.143036 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-19.ec2.internal" event="NodeHasSufficientMemory" Jul 07 14:34:19.143127 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:19.143062 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-19.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 14:34:19.143127 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:19.143074 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-19.ec2.internal" event="NodeHasSufficientPID" Jul 07 14:34:19.155308 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:19.155289 2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-19.ec2.internal\" not found" node="ip-10-0-136-19.ec2.internal" Jul 07 14:34:19.158805 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:19.158789 2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-19.ec2.internal\" not found" node="ip-10-0-136-19.ec2.internal" Jul 07 14:34:19.167559 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:19.167541 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ff9a5e8afa26523332eda2af110e42ee-config\") pod \"kube-apiserver-proxy-ip-10-0-136-19.ec2.internal\" (UID: \"ff9a5e8afa26523332eda2af110e42ee\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-19.ec2.internal" Jul 07 14:34:19.167646 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:19.167564 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/340cb4146c128f06bd339f0108cd15f2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-19.ec2.internal\" (UID: \"340cb4146c128f06bd339f0108cd15f2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-19.ec2.internal" Jul 07 14:34:19.167646 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:19.167581 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/340cb4146c128f06bd339f0108cd15f2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-19.ec2.internal\" (UID: \"340cb4146c128f06bd339f0108cd15f2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-19.ec2.internal" Jul 07 14:34:19.195763 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:19.195725 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-19.ec2.internal\" not found" Jul 07 14:34:19.267923 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:19.267900 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ff9a5e8afa26523332eda2af110e42ee-config\") pod \"kube-apiserver-proxy-ip-10-0-136-19.ec2.internal\" (UID: \"ff9a5e8afa26523332eda2af110e42ee\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-19.ec2.internal" Jul 07 14:34:19.268070 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:19.267928 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/340cb4146c128f06bd339f0108cd15f2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-19.ec2.internal\" (UID: \"340cb4146c128f06bd339f0108cd15f2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-19.ec2.internal" Jul 07 14:34:19.268070 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:19.267948 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/340cb4146c128f06bd339f0108cd15f2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-19.ec2.internal\" (UID: \"340cb4146c128f06bd339f0108cd15f2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-19.ec2.internal" Jul 07 14:34:19.268070 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:19.267996 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/340cb4146c128f06bd339f0108cd15f2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-19.ec2.internal\" (UID: \"340cb4146c128f06bd339f0108cd15f2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-19.ec2.internal" Jul 07 14:34:19.268070 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:19.268001 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ff9a5e8afa26523332eda2af110e42ee-config\") pod \"kube-apiserver-proxy-ip-10-0-136-19.ec2.internal\" (UID: \"ff9a5e8afa26523332eda2af110e42ee\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-19.ec2.internal" Jul 07 14:34:19.268070 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:19.268005 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/340cb4146c128f06bd339f0108cd15f2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-19.ec2.internal\" (UID: \"340cb4146c128f06bd339f0108cd15f2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-19.ec2.internal" Jul 07 14:34:19.296089 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:19.296018 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-19.ec2.internal\" not found" Jul 07 14:34:19.398274 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:19.398215 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-19.ec2.internal\" not found" Jul 07 14:34:19.459878 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:19.459841 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-19.ec2.internal" Jul 07 14:34:19.463234 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:19.463217 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-19.ec2.internal" Jul 07 14:34:19.499117 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:19.499075 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-19.ec2.internal\" not found" Jul 07 14:34:19.601273 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:19.601225 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-19.ec2.internal\" not found" Jul 07 14:34:19.701423 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:19.701376 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-19.ec2.internal\" not found" Jul 07 14:34:19.803815 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:19.803783 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-19.ec2.internal\" not found" Jul 07 14:34:19.810446 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:19.810417 2555 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 07 14:34:19.810630 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:19.810612 2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 07 14:34:19.888683 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:19.888593 2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 14:29:19 +0000 UTC" deadline="2028-03-02 01:10:06.3279528 +0000 UTC" Jul 07 14:34:19.888683 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:19.888627 2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14482h35m46.43932928s" Jul 07 14:34:19.904794 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:19.904766 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-19.ec2.internal\" not found" Jul 07 14:34:20.004986 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:20.004957 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-19.ec2.internal\" not found" Jul 07 14:34:20.044433 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.044403 2555 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 14:34:20.105088 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:20.105049 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-19.ec2.internal\" not found" Jul 07 14:34:20.168228 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:20.168184 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod340cb4146c128f06bd339f0108cd15f2.slice/crio-aba537662881c9378c1c78bf2d66b87b0d58a67cc7e5744b401dc1d5d3eca4ad WatchSource:0}: Error finding container aba537662881c9378c1c78bf2d66b87b0d58a67cc7e5744b401dc1d5d3eca4ad: Status 404 returned error can't find the container with id aba537662881c9378c1c78bf2d66b87b0d58a67cc7e5744b401dc1d5d3eca4ad Jul 07 14:34:20.172598 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.172576 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 14:34:20.179212 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:20.179186 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff9a5e8afa26523332eda2af110e42ee.slice/crio-d44703a5ba155230b5e6c2a656fce228c34443282c0feb80f02afdc5f762b476 WatchSource:0}: Error finding container d44703a5ba155230b5e6c2a656fce228c34443282c0feb80f02afdc5f762b476: Status 404 returned error can't find the container with id d44703a5ba155230b5e6c2a656fce228c34443282c0feb80f02afdc5f762b476 Jul 07 14:34:20.205736 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:20.205706 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-19.ec2.internal\" not found" Jul 07 14:34:20.286225 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.286198 2555 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 14:34:20.306823 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:20.306794 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-19.ec2.internal\" not found" Jul 07 14:34:20.327998 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.327974 2555 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 14:34:20.365656 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.365622 2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-19.ec2.internal" Jul 07 14:34:20.377255 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.377232 2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 14:34:20.378429 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.378415 2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-19.ec2.internal" Jul 07 14:34:20.387422 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.387403 2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 14:34:20.846266 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.846043 2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 14:34:20.851830 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.851805 2555 apiserver.go:52] "Watching apiserver" Jul 07 14:34:20.859860 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.859455 2555 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 07 14:34:20.861732 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.861257 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-9t7xp","openshift-multus/multus-bfcr2","openshift-multus/network-metrics-daemon-lpmtl","kube-system/global-pull-secret-syncer-bq6hn","kube-system/konnectivity-agent-4zclx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb4h5","openshift-cluster-node-tuning-operator/tuned-vs7xn","openshift-network-diagnostics/network-check-target-zsq7v","openshift-network-operator/iptables-alerter-mj6xk","openshift-ovn-kubernetes/ovnkube-node-59hn4","kube-system/kube-apiserver-proxy-ip-10-0-136-19.ec2.internal","openshift-dns/node-resolver-czppd","openshift-image-registry/node-ca-vkwws","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-19.ec2.internal"] Jul 07 14:34:20.863997 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.863971 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4zclx" Jul 07 14:34:20.867880 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.866577 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" Jul 07 14:34:20.867880 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.867284 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 07 14:34:20.867880 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.867545 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-m5k5g\"" Jul 07 14:34:20.867880 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.867582 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 07 14:34:20.867880 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.867780 2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 07 14:34:20.868833 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.868805 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bq6hn" Jul 07 14:34:20.868953 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:20.868883 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bq6hn" podUID="686ecb3c-7c9a-4ac1-ac27-4430651b0870" Jul 07 14:34:20.869148 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.869128 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 07 14:34:20.869216 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.869182 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 07 14:34:20.869495 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.869465 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2sk6v\"" Jul 07 14:34:20.871788 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.871311 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9t7xp" Jul 07 14:34:20.874063 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.873950 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpmtl" Jul 07 14:34:20.874063 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:20.874027 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpmtl" podUID="e3bbd20c-73f5-4d24-aed9-145e4b89a590" Jul 07 14:34:20.874860 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.874505 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 07 14:34:20.874860 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.874537 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 07 14:34:20.874860 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.874549 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 07 14:34:20.874860 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.874806 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 07 14:34:20.874860 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.874811 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 07 14:34:20.875304 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.875162 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-f62zn\"" Jul 07 14:34:20.877110 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.876880 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb4h5" Jul 07 14:34:20.877456 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.877433 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aaf88254-ff84-430f-b609-debc1b4b170c-host\") pod \"tuned-vs7xn\" (UID: \"aaf88254-ff84-430f-b609-debc1b4b170c\") " pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" Jul 07 14:34:20.877542 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.877474 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/aaf88254-ff84-430f-b609-debc1b4b170c-etc-tuned\") pod \"tuned-vs7xn\" (UID: \"aaf88254-ff84-430f-b609-debc1b4b170c\") " pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" Jul 07 14:34:20.877542 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.877502 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpzbh\" (UniqueName: \"kubernetes.io/projected/aaf88254-ff84-430f-b609-debc1b4b170c-kube-api-access-lpzbh\") pod \"tuned-vs7xn\" (UID: \"aaf88254-ff84-430f-b609-debc1b4b170c\") " pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" Jul 07 14:34:20.877542 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.877528 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3ea25b41-29d9-4e81-b4d5-6252d76a8e66-konnectivity-ca\") pod \"konnectivity-agent-4zclx\" (UID: \"3ea25b41-29d9-4e81-b4d5-6252d76a8e66\") " pod="kube-system/konnectivity-agent-4zclx" Jul 07 14:34:20.877703 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.877551 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/aaf88254-ff84-430f-b609-debc1b4b170c-etc-sysconfig\") pod \"tuned-vs7xn\" (UID: \"aaf88254-ff84-430f-b609-debc1b4b170c\") " pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" Jul 07 14:34:20.877703 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.877611 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/aaf88254-ff84-430f-b609-debc1b4b170c-etc-sysctl-d\") pod \"tuned-vs7xn\" (UID: \"aaf88254-ff84-430f-b609-debc1b4b170c\") " pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" Jul 07 14:34:20.877703 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.877637 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/aaf88254-ff84-430f-b609-debc1b4b170c-run\") pod \"tuned-vs7xn\" (UID: \"aaf88254-ff84-430f-b609-debc1b4b170c\") " pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" Jul 07 14:34:20.877703 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.877662 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aaf88254-ff84-430f-b609-debc1b4b170c-sys\") pod \"tuned-vs7xn\" (UID: \"aaf88254-ff84-430f-b609-debc1b4b170c\") " pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" Jul 07 14:34:20.877703 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.877687 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3ea25b41-29d9-4e81-b4d5-6252d76a8e66-agent-certs\") pod \"konnectivity-agent-4zclx\" (UID: \"3ea25b41-29d9-4e81-b4d5-6252d76a8e66\") " pod="kube-system/konnectivity-agent-4zclx" Jul 07 14:34:20.877975 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.877730 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/aaf88254-ff84-430f-b609-debc1b4b170c-etc-sysctl-conf\") pod \"tuned-vs7xn\" (UID: \"aaf88254-ff84-430f-b609-debc1b4b170c\") " pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" Jul 07 14:34:20.877975 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.877800 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/aaf88254-ff84-430f-b609-debc1b4b170c-etc-systemd\") pod \"tuned-vs7xn\" (UID: \"aaf88254-ff84-430f-b609-debc1b4b170c\") " pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" Jul 07 14:34:20.877975 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.877883 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aaf88254-ff84-430f-b609-debc1b4b170c-tmp\") pod \"tuned-vs7xn\" (UID: \"aaf88254-ff84-430f-b609-debc1b4b170c\") " pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" Jul 07 14:34:20.877975 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.877936 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/aaf88254-ff84-430f-b609-debc1b4b170c-etc-modprobe-d\") pod \"tuned-vs7xn\" (UID: \"aaf88254-ff84-430f-b609-debc1b4b170c\") " pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" Jul 07 14:34:20.877975 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.877970 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aaf88254-ff84-430f-b609-debc1b4b170c-etc-kubernetes\") pod \"tuned-vs7xn\" (UID: \"aaf88254-ff84-430f-b609-debc1b4b170c\") " pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" Jul 07 14:34:20.878212 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.878011 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aaf88254-ff84-430f-b609-debc1b4b170c-lib-modules\") pod \"tuned-vs7xn\" (UID: \"aaf88254-ff84-430f-b609-debc1b4b170c\") " pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" Jul 07 14:34:20.878212 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.878044 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aaf88254-ff84-430f-b609-debc1b4b170c-var-lib-kubelet\") pod \"tuned-vs7xn\" (UID: \"aaf88254-ff84-430f-b609-debc1b4b170c\") " pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" Jul 07 14:34:20.879694 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.879671 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 07 14:34:20.881156 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.880103 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-sltn4\"" Jul 07 14:34:20.881156 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.880471 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 07 14:34:20.881156 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.880700 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bfcr2" Jul 07 14:34:20.881987 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.881706 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 07 14:34:20.882387 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.882331 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zsq7v" Jul 07 14:34:20.883440 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:20.883406 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zsq7v" podUID="3e0c7b26-181b-47f2-8503-b66092a86230" Jul 07 14:34:20.886987 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.884283 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-clmnv\"" Jul 07 14:34:20.886987 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.884346 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 07 14:34:20.886987 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.885125 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mj6xk" Jul 07 14:34:20.890129 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.888509 2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 14:34:20.890129 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.888740 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rxtbv\"" Jul 07 14:34:20.890129 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.888814 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 07 14:34:20.890129 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.889116 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 07 14:34:20.890129 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.889176 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 07 14:34:20.890129 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.889273 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:20.892051 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.891977 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 07 14:34:20.892359 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.892338 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 07 14:34:20.892556 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.892536 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 07 14:34:20.892735 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.892704 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9kqrk\"" Jul 07 14:34:20.892735 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.892712 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 07 14:34:20.893365 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.893342 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 07 14:34:20.893621 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.893599 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 07 14:34:20.893721 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.893622 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-czppd" Jul 07 14:34:20.896201 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.896180 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-q68z7\"" Jul 07 14:34:20.896300 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.896286 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vkwws" Jul 07 14:34:20.896489 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.896470 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 07 14:34:20.896742 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.896726 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 07 14:34:20.899284 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.899262 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 07 14:34:20.899658 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.899640 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 07 14:34:20.899738 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.899661 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 07 14:34:20.899807 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.899785 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zrghk\"" Jul 07 14:34:20.908986 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.908958 2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7xhdr" Jul 07 14:34:20.917295 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.917276 2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7xhdr" Jul 07 14:34:20.966497 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.966470 2555 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 07 14:34:20.979282 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.979246 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/686ecb3c-7c9a-4ac1-ac27-4430651b0870-dbus\") pod \"global-pull-secret-syncer-bq6hn\" (UID: \"686ecb3c-7c9a-4ac1-ac27-4430651b0870\") " pod="kube-system/global-pull-secret-syncer-bq6hn" Jul 07 14:34:20.979282 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.979292 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8lcx\" (UniqueName: \"kubernetes.io/projected/3e0c7b26-181b-47f2-8503-b66092a86230-kube-api-access-r8lcx\") pod \"network-check-target-zsq7v\" (UID: \"3e0c7b26-181b-47f2-8503-b66092a86230\") " pod="openshift-network-diagnostics/network-check-target-zsq7v" Jul 07 14:34:20.979573 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.979326 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/aaf88254-ff84-430f-b609-debc1b4b170c-etc-sysconfig\") pod \"tuned-vs7xn\" (UID: \"aaf88254-ff84-430f-b609-debc1b4b170c\") " pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" Jul 07 14:34:20.979573 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.979350 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/303d9d00-aaf9-431e-9980-60fd60904fad-host-run-ovn-kubernetes\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:20.979573 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.979373 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/303d9d00-aaf9-431e-9980-60fd60904fad-host-kubelet\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:20.979573 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.979397 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/303d9d00-aaf9-431e-9980-60fd60904fad-run-systemd\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:20.979573 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.979422 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/303d9d00-aaf9-431e-9980-60fd60904fad-run-openvswitch\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:20.979573 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.979446 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/303d9d00-aaf9-431e-9980-60fd60904fad-host-cni-netd\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:20.979573 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.979468 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ff6cf858-eb9e-4adc-b4ae-9671841ef488-sys-fs\") pod \"aws-ebs-csi-driver-node-sb4h5\" (UID: \"ff6cf858-eb9e-4adc-b4ae-9671841ef488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb4h5" Jul 07 14:34:20.979573 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.979496 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/aaf88254-ff84-430f-b609-debc1b4b170c-etc-sysctl-conf\") pod \"tuned-vs7xn\" (UID: \"aaf88254-ff84-430f-b609-debc1b4b170c\") " pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" Jul 07 14:34:20.979573 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.979518 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aaf88254-ff84-430f-b609-debc1b4b170c-tmp\") pod \"tuned-vs7xn\" (UID: \"aaf88254-ff84-430f-b609-debc1b4b170c\") " pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" Jul 07 14:34:20.979573 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.979522 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/aaf88254-ff84-430f-b609-debc1b4b170c-etc-sysconfig\") pod \"tuned-vs7xn\" (UID: \"aaf88254-ff84-430f-b609-debc1b4b170c\") " pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" Jul 07 14:34:20.979573 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.979543 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89159896-e9f7-42e0-93e7-23122a296d8c-system-cni-dir\") pod \"multus-additional-cni-plugins-9t7xp\" (UID: \"89159896-e9f7-42e0-93e7-23122a296d8c\") " pod="openshift-multus/multus-additional-cni-plugins-9t7xp" Jul 07 14:34:20.980178 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.979569 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89159896-e9f7-42e0-93e7-23122a296d8c-cnibin\") pod \"multus-additional-cni-plugins-9t7xp\" (UID: \"89159896-e9f7-42e0-93e7-23122a296d8c\") " pod="openshift-multus/multus-additional-cni-plugins-9t7xp" Jul 07 14:34:20.980178 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.979630 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/89159896-e9f7-42e0-93e7-23122a296d8c-os-release\") pod \"multus-additional-cni-plugins-9t7xp\" (UID: \"89159896-e9f7-42e0-93e7-23122a296d8c\") " pod="openshift-multus/multus-additional-cni-plugins-9t7xp" Jul 07 14:34:20.980178 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.979658 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9ptv\" (UniqueName: \"kubernetes.io/projected/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-kube-api-access-k9ptv\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:20.980178 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.979683 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7d94747d-5e8a-4eab-8234-969f7dfbaee5-serviceca\") pod \"node-ca-vkwws\" (UID: \"7d94747d-5e8a-4eab-8234-969f7dfbaee5\") " pod="openshift-image-registry/node-ca-vkwws" Jul 07 14:34:20.980178 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.979707 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/aaf88254-ff84-430f-b609-debc1b4b170c-etc-modprobe-d\") pod \"tuned-vs7xn\" (UID: \"aaf88254-ff84-430f-b609-debc1b4b170c\") " pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" Jul 07 14:34:20.980178 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.979733 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/303d9d00-aaf9-431e-9980-60fd60904fad-log-socket\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:20.980178 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.979756 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/303d9d00-aaf9-431e-9980-60fd60904fad-host-cni-bin\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:20.980178 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.979780 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/553f749a-649c-41c0-aa39-a0cb13725ca8-tmp-dir\") pod \"node-resolver-czppd\" (UID: \"553f749a-649c-41c0-aa39-a0cb13725ca8\") " pod="openshift-dns/node-resolver-czppd" Jul 07 14:34:20.980178 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.979867 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89159896-e9f7-42e0-93e7-23122a296d8c-cni-binary-copy\") pod \"multus-additional-cni-plugins-9t7xp\" (UID: \"89159896-e9f7-42e0-93e7-23122a296d8c\") " pod="openshift-multus/multus-additional-cni-plugins-9t7xp" Jul 07 14:34:20.980178 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.979898 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-host-var-lib-cni-multus\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:20.980178 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.979924 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-host-var-lib-kubelet\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:20.980178 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.979949 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-multus-daemon-config\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:20.980178 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.979984 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ff6cf858-eb9e-4adc-b4ae-9671841ef488-socket-dir\") pod \"aws-ebs-csi-driver-node-sb4h5\" (UID: \"ff6cf858-eb9e-4adc-b4ae-9671841ef488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb4h5" Jul 07 14:34:20.980178 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.980014 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/aaf88254-ff84-430f-b609-debc1b4b170c-etc-sysctl-d\") pod \"tuned-vs7xn\" (UID: \"aaf88254-ff84-430f-b609-debc1b4b170c\") " pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" Jul 07 14:34:20.980178 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.980038 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/aaf88254-ff84-430f-b609-debc1b4b170c-run\") pod \"tuned-vs7xn\" (UID: \"aaf88254-ff84-430f-b609-debc1b4b170c\") " pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" Jul 07 14:34:20.980178 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.980063 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/303d9d00-aaf9-431e-9980-60fd60904fad-node-log\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:20.980178 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.980016 2555 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 07 14:34:20.981098 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.980090 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/553f749a-649c-41c0-aa39-a0cb13725ca8-hosts-file\") pod \"node-resolver-czppd\" (UID: \"553f749a-649c-41c0-aa39-a0cb13725ca8\") " pod="openshift-dns/node-resolver-czppd" Jul 07 14:34:20.981098 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.980134 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g29pn\" (UniqueName: \"kubernetes.io/projected/553f749a-649c-41c0-aa39-a0cb13725ca8-kube-api-access-g29pn\") pod \"node-resolver-czppd\" (UID: \"553f749a-649c-41c0-aa39-a0cb13725ca8\") " pod="openshift-dns/node-resolver-czppd" Jul 07 14:34:20.981098 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.980190 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ff6cf858-eb9e-4adc-b4ae-9671841ef488-registration-dir\") pod \"aws-ebs-csi-driver-node-sb4h5\" (UID: \"ff6cf858-eb9e-4adc-b4ae-9671841ef488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb4h5" Jul 07 14:34:20.981098 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.980215 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ff6cf858-eb9e-4adc-b4ae-9671841ef488-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sb4h5\" (UID: \"ff6cf858-eb9e-4adc-b4ae-9671841ef488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb4h5" Jul 07 14:34:20.981098 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.980246 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-os-release\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:20.981098 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.980275 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-host-var-lib-cni-bin\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:20.981098 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.980284 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/aaf88254-ff84-430f-b609-debc1b4b170c-etc-modprobe-d\") pod \"tuned-vs7xn\" (UID: \"aaf88254-ff84-430f-b609-debc1b4b170c\") " pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" Jul 07 14:34:20.981098 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.980313 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/686ecb3c-7c9a-4ac1-ac27-4430651b0870-kubelet-config\") pod \"global-pull-secret-syncer-bq6hn\" (UID: \"686ecb3c-7c9a-4ac1-ac27-4430651b0870\") " pod="kube-system/global-pull-secret-syncer-bq6hn" Jul 07 14:34:20.981098 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.980344 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/303d9d00-aaf9-431e-9980-60fd60904fad-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:20.981098 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.980371 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89159896-e9f7-42e0-93e7-23122a296d8c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9t7xp\" (UID: \"89159896-e9f7-42e0-93e7-23122a296d8c\") " pod="openshift-multus/multus-additional-cni-plugins-9t7xp" Jul 07 14:34:20.981098 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.980375 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/aaf88254-ff84-430f-b609-debc1b4b170c-run\") pod \"tuned-vs7xn\" (UID: \"aaf88254-ff84-430f-b609-debc1b4b170c\") " pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" Jul 07 14:34:20.981098 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.980414 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ff6cf858-eb9e-4adc-b4ae-9671841ef488-device-dir\") pod \"aws-ebs-csi-driver-node-sb4h5\" (UID: \"ff6cf858-eb9e-4adc-b4ae-9671841ef488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb4h5" Jul 07 14:34:20.981098 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.980538 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-multus-socket-dir-parent\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:20.981098 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.980587 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aaf88254-ff84-430f-b609-debc1b4b170c-etc-kubernetes\") pod \"tuned-vs7xn\" (UID: \"aaf88254-ff84-430f-b609-debc1b4b170c\") " pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" Jul 07 14:34:20.981098 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.980601 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/aaf88254-ff84-430f-b609-debc1b4b170c-etc-sysctl-d\") pod \"tuned-vs7xn\" (UID: \"aaf88254-ff84-430f-b609-debc1b4b170c\") " pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" Jul 07 14:34:20.981098 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.980634 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/303d9d00-aaf9-431e-9980-60fd60904fad-ovnkube-config\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:20.982003 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.980678 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8pp8\" (UniqueName: \"kubernetes.io/projected/e3bbd20c-73f5-4d24-aed9-145e4b89a590-kube-api-access-w8pp8\") pod \"network-metrics-daemon-lpmtl\" (UID: \"e3bbd20c-73f5-4d24-aed9-145e4b89a590\") " pod="openshift-multus/network-metrics-daemon-lpmtl" Jul 07 14:34:20.982003 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.980704 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-system-cni-dir\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:20.982003 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.980731 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-etc-kubernetes\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:20.982003 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.980741 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/aaf88254-ff84-430f-b609-debc1b4b170c-etc-sysctl-conf\") pod \"tuned-vs7xn\" (UID: \"aaf88254-ff84-430f-b609-debc1b4b170c\") " pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" Jul 07 14:34:20.982003 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.980758 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3ea25b41-29d9-4e81-b4d5-6252d76a8e66-konnectivity-ca\") pod \"konnectivity-agent-4zclx\" (UID: \"3ea25b41-29d9-4e81-b4d5-6252d76a8e66\") " pod="kube-system/konnectivity-agent-4zclx" Jul 07 14:34:20.982003 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.980790 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aaf88254-ff84-430f-b609-debc1b4b170c-etc-kubernetes\") pod \"tuned-vs7xn\" (UID: \"aaf88254-ff84-430f-b609-debc1b4b170c\") " pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" Jul 07 14:34:20.982003 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.980846 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/303d9d00-aaf9-431e-9980-60fd60904fad-ovn-node-metrics-cert\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:20.982003 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.980881 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-cnibin\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:20.982003 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.980923 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-host-run-netns\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:20.982003 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.980952 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/303d9d00-aaf9-431e-9980-60fd60904fad-host-run-netns\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:20.982003 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.980989 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/686ecb3c-7c9a-4ac1-ac27-4430651b0870-original-pull-secret\") pod \"global-pull-secret-syncer-bq6hn\" (UID: \"686ecb3c-7c9a-4ac1-ac27-4430651b0870\") " pod="kube-system/global-pull-secret-syncer-bq6hn" Jul 07 14:34:20.982003 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.981023 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/aaf88254-ff84-430f-b609-debc1b4b170c-etc-systemd\") pod \"tuned-vs7xn\" (UID: \"aaf88254-ff84-430f-b609-debc1b4b170c\") " pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" Jul 07 14:34:20.982003 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.981074 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/aaf88254-ff84-430f-b609-debc1b4b170c-etc-systemd\") pod \"tuned-vs7xn\" (UID: \"aaf88254-ff84-430f-b609-debc1b4b170c\") " pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" Jul 07 14:34:20.982003 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.981111 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/303d9d00-aaf9-431e-9980-60fd60904fad-systemd-units\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:20.982003 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.981152 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/303d9d00-aaf9-431e-9980-60fd60904fad-host-slash\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:20.982003 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.981187 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/89159896-e9f7-42e0-93e7-23122a296d8c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9t7xp\" (UID: \"89159896-e9f7-42e0-93e7-23122a296d8c\") " pod="openshift-multus/multus-additional-cni-plugins-9t7xp" Jul 07 14:34:20.982003 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.981218 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3bbd20c-73f5-4d24-aed9-145e4b89a590-metrics-certs\") pod \"network-metrics-daemon-lpmtl\" (UID: \"e3bbd20c-73f5-4d24-aed9-145e4b89a590\") " pod="openshift-multus/network-metrics-daemon-lpmtl" Jul 07 14:34:20.982905 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.981244 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c06aa1d3-1eb1-4a64-9677-2b7328a9cff9-host-slash\") pod \"iptables-alerter-mj6xk\" (UID: \"c06aa1d3-1eb1-4a64-9677-2b7328a9cff9\") " pod="openshift-network-operator/iptables-alerter-mj6xk" Jul 07 14:34:20.982905 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.981271 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgnsk\" (UniqueName: \"kubernetes.io/projected/c06aa1d3-1eb1-4a64-9677-2b7328a9cff9-kube-api-access-mgnsk\") pod \"iptables-alerter-mj6xk\" (UID: \"c06aa1d3-1eb1-4a64-9677-2b7328a9cff9\") " pod="openshift-network-operator/iptables-alerter-mj6xk" Jul 07 14:34:20.982905 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.981302 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aaf88254-ff84-430f-b609-debc1b4b170c-lib-modules\") pod \"tuned-vs7xn\" (UID: \"aaf88254-ff84-430f-b609-debc1b4b170c\") " pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" Jul 07 14:34:20.982905 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.981329 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aaf88254-ff84-430f-b609-debc1b4b170c-host\") pod \"tuned-vs7xn\" (UID: \"aaf88254-ff84-430f-b609-debc1b4b170c\") " pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" Jul 07 14:34:20.982905 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.981352 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/aaf88254-ff84-430f-b609-debc1b4b170c-etc-tuned\") pod \"tuned-vs7xn\" (UID: \"aaf88254-ff84-430f-b609-debc1b4b170c\") " pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" Jul 07 14:34:20.982905 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.981379 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-hostroot\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:20.982905 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.981404 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c06aa1d3-1eb1-4a64-9677-2b7328a9cff9-iptables-alerter-script\") pod \"iptables-alerter-mj6xk\" (UID: \"c06aa1d3-1eb1-4a64-9677-2b7328a9cff9\") " pod="openshift-network-operator/iptables-alerter-mj6xk" Jul 07 14:34:20.982905 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.981430 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/303d9d00-aaf9-431e-9980-60fd60904fad-etc-openvswitch\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:20.982905 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.981456 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8srjl\" (UniqueName: \"kubernetes.io/projected/303d9d00-aaf9-431e-9980-60fd60904fad-kube-api-access-8srjl\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:20.982905 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.981465 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aaf88254-ff84-430f-b609-debc1b4b170c-lib-modules\") pod \"tuned-vs7xn\" (UID: \"aaf88254-ff84-430f-b609-debc1b4b170c\") " pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" Jul 07 14:34:20.982905 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.981479 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-multus-cni-dir\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:20.982905 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.981506 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-multus-conf-dir\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:20.982905 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.981565 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlmm6\" (UniqueName: \"kubernetes.io/projected/7d94747d-5e8a-4eab-8234-969f7dfbaee5-kube-api-access-qlmm6\") pod \"node-ca-vkwws\" (UID: \"7d94747d-5e8a-4eab-8234-969f7dfbaee5\") " pod="openshift-image-registry/node-ca-vkwws" Jul 07 14:34:20.982905 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.981582 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aaf88254-ff84-430f-b609-debc1b4b170c-host\") pod \"tuned-vs7xn\" (UID: \"aaf88254-ff84-430f-b609-debc1b4b170c\") " pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" Jul 07 14:34:20.982905 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.981597 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/303d9d00-aaf9-431e-9980-60fd60904fad-var-lib-openvswitch\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:20.982905 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.981651 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/303d9d00-aaf9-431e-9980-60fd60904fad-run-ovn\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:20.982905 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.981650 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3ea25b41-29d9-4e81-b4d5-6252d76a8e66-konnectivity-ca\") pod \"konnectivity-agent-4zclx\" (UID: \"3ea25b41-29d9-4e81-b4d5-6252d76a8e66\") " pod="kube-system/konnectivity-agent-4zclx" Jul 07 14:34:20.983844 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.981702 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/89159896-e9f7-42e0-93e7-23122a296d8c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9t7xp\" (UID: \"89159896-e9f7-42e0-93e7-23122a296d8c\") " pod="openshift-multus/multus-additional-cni-plugins-9t7xp" Jul 07 14:34:20.983844 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.981746 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmzxg\" (UniqueName: \"kubernetes.io/projected/ff6cf858-eb9e-4adc-b4ae-9671841ef488-kube-api-access-lmzxg\") pod \"aws-ebs-csi-driver-node-sb4h5\" (UID: \"ff6cf858-eb9e-4adc-b4ae-9671841ef488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb4h5" Jul 07 14:34:20.983844 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.981779 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-host-run-multus-certs\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:20.983844 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.981806 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7d94747d-5e8a-4eab-8234-969f7dfbaee5-host\") pod \"node-ca-vkwws\" (UID: \"7d94747d-5e8a-4eab-8234-969f7dfbaee5\") " pod="openshift-image-registry/node-ca-vkwws" Jul 07 14:34:20.983844 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.981878 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aaf88254-ff84-430f-b609-debc1b4b170c-sys\") pod \"tuned-vs7xn\" (UID: \"aaf88254-ff84-430f-b609-debc1b4b170c\") " pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" Jul 07 14:34:20.983844 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.981941 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aaf88254-ff84-430f-b609-debc1b4b170c-sys\") pod \"tuned-vs7xn\" (UID: \"aaf88254-ff84-430f-b609-debc1b4b170c\") " pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" Jul 07 14:34:20.983844 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.981944 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3ea25b41-29d9-4e81-b4d5-6252d76a8e66-agent-certs\") pod \"konnectivity-agent-4zclx\" (UID: \"3ea25b41-29d9-4e81-b4d5-6252d76a8e66\") " pod="kube-system/konnectivity-agent-4zclx" Jul 07 14:34:20.983844 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.981981 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/303d9d00-aaf9-431e-9980-60fd60904fad-env-overrides\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:20.983844 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.982004 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ff6cf858-eb9e-4adc-b4ae-9671841ef488-etc-selinux\") pod \"aws-ebs-csi-driver-node-sb4h5\" (UID: \"ff6cf858-eb9e-4adc-b4ae-9671841ef488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb4h5" Jul 07 14:34:20.983844 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.982103 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-cni-binary-copy\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:20.983844 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.982132 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aaf88254-ff84-430f-b609-debc1b4b170c-var-lib-kubelet\") pod \"tuned-vs7xn\" (UID: \"aaf88254-ff84-430f-b609-debc1b4b170c\") " pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" Jul 07 14:34:20.983844 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.982166 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpzbh\" (UniqueName: \"kubernetes.io/projected/aaf88254-ff84-430f-b609-debc1b4b170c-kube-api-access-lpzbh\") pod \"tuned-vs7xn\" (UID: \"aaf88254-ff84-430f-b609-debc1b4b170c\") " pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" Jul 07 14:34:20.983844 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.982210 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aaf88254-ff84-430f-b609-debc1b4b170c-var-lib-kubelet\") pod \"tuned-vs7xn\" (UID: \"aaf88254-ff84-430f-b609-debc1b4b170c\") " pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" Jul 07 14:34:20.983844 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.982254 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/303d9d00-aaf9-431e-9980-60fd60904fad-ovnkube-script-lib\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:20.983844 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.982377 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmb7b\" (UniqueName: \"kubernetes.io/projected/89159896-e9f7-42e0-93e7-23122a296d8c-kube-api-access-dmb7b\") pod \"multus-additional-cni-plugins-9t7xp\" (UID: \"89159896-e9f7-42e0-93e7-23122a296d8c\") " pod="openshift-multus/multus-additional-cni-plugins-9t7xp" Jul 07 14:34:20.983844 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.982407 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-host-run-k8s-cni-cncf-io\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:20.985801 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.985744 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aaf88254-ff84-430f-b609-debc1b4b170c-tmp\") pod \"tuned-vs7xn\" (UID: \"aaf88254-ff84-430f-b609-debc1b4b170c\") " pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" Jul 07 14:34:20.985936 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.985845 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/aaf88254-ff84-430f-b609-debc1b4b170c-etc-tuned\") pod \"tuned-vs7xn\" (UID: \"aaf88254-ff84-430f-b609-debc1b4b170c\") " pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" Jul 07 14:34:20.987151 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.987124 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3ea25b41-29d9-4e81-b4d5-6252d76a8e66-agent-certs\") pod \"konnectivity-agent-4zclx\" (UID: \"3ea25b41-29d9-4e81-b4d5-6252d76a8e66\") " pod="kube-system/konnectivity-agent-4zclx" Jul 07 14:34:20.996086 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:20.996058 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpzbh\" (UniqueName: \"kubernetes.io/projected/aaf88254-ff84-430f-b609-debc1b4b170c-kube-api-access-lpzbh\") pod \"tuned-vs7xn\" (UID: \"aaf88254-ff84-430f-b609-debc1b4b170c\") " pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" Jul 07 14:34:21.045108 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.045048 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-19.ec2.internal" event={"ID":"ff9a5e8afa26523332eda2af110e42ee","Type":"ContainerStarted","Data":"d44703a5ba155230b5e6c2a656fce228c34443282c0feb80f02afdc5f762b476"} Jul 07 14:34:21.046441 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.046404 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-19.ec2.internal" event={"ID":"340cb4146c128f06bd339f0108cd15f2","Type":"ContainerStarted","Data":"aba537662881c9378c1c78bf2d66b87b0d58a67cc7e5744b401dc1d5d3eca4ad"} Jul 07 14:34:21.082942 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.082895 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/303d9d00-aaf9-431e-9980-60fd60904fad-ovn-node-metrics-cert\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:21.082942 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.082944 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-cnibin\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:21.083176 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.082972 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-host-run-netns\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:21.083176 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.082997 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/303d9d00-aaf9-431e-9980-60fd60904fad-host-run-netns\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:21.083176 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.083024 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/686ecb3c-7c9a-4ac1-ac27-4430651b0870-original-pull-secret\") pod \"global-pull-secret-syncer-bq6hn\" (UID: \"686ecb3c-7c9a-4ac1-ac27-4430651b0870\") " pod="kube-system/global-pull-secret-syncer-bq6hn" Jul 07 14:34:21.083176 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.083050 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/303d9d00-aaf9-431e-9980-60fd60904fad-systemd-units\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:21.083176 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.083074 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/303d9d00-aaf9-431e-9980-60fd60904fad-host-slash\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:21.083176 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.083083 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-cnibin\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:21.083176 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.083098 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/89159896-e9f7-42e0-93e7-23122a296d8c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9t7xp\" (UID: \"89159896-e9f7-42e0-93e7-23122a296d8c\") " pod="openshift-multus/multus-additional-cni-plugins-9t7xp" Jul 07 14:34:21.083176 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.083146 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3bbd20c-73f5-4d24-aed9-145e4b89a590-metrics-certs\") pod \"network-metrics-daemon-lpmtl\" (UID: \"e3bbd20c-73f5-4d24-aed9-145e4b89a590\") " pod="openshift-multus/network-metrics-daemon-lpmtl" Jul 07 14:34:21.083176 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.083178 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c06aa1d3-1eb1-4a64-9677-2b7328a9cff9-host-slash\") pod \"iptables-alerter-mj6xk\" (UID: \"c06aa1d3-1eb1-4a64-9677-2b7328a9cff9\") " pod="openshift-network-operator/iptables-alerter-mj6xk" Jul 07 14:34:21.083524 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.083209 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgnsk\" (UniqueName: \"kubernetes.io/projected/c06aa1d3-1eb1-4a64-9677-2b7328a9cff9-kube-api-access-mgnsk\") pod \"iptables-alerter-mj6xk\" (UID: \"c06aa1d3-1eb1-4a64-9677-2b7328a9cff9\") " pod="openshift-network-operator/iptables-alerter-mj6xk" Jul 07 14:34:21.083524 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.083239 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-hostroot\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:21.083524 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.083267 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c06aa1d3-1eb1-4a64-9677-2b7328a9cff9-iptables-alerter-script\") pod \"iptables-alerter-mj6xk\" (UID: \"c06aa1d3-1eb1-4a64-9677-2b7328a9cff9\") " pod="openshift-network-operator/iptables-alerter-mj6xk" Jul 07 14:34:21.083524 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.083293 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/303d9d00-aaf9-431e-9980-60fd60904fad-etc-openvswitch\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:21.083524 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.083319 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8srjl\" (UniqueName: \"kubernetes.io/projected/303d9d00-aaf9-431e-9980-60fd60904fad-kube-api-access-8srjl\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:21.083524 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.083345 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-multus-cni-dir\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:21.083524 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.083368 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-multus-conf-dir\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:21.083524 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.083393 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlmm6\" (UniqueName: \"kubernetes.io/projected/7d94747d-5e8a-4eab-8234-969f7dfbaee5-kube-api-access-qlmm6\") pod \"node-ca-vkwws\" (UID: \"7d94747d-5e8a-4eab-8234-969f7dfbaee5\") " pod="openshift-image-registry/node-ca-vkwws" Jul 07 14:34:21.083524 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.083413 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-hostroot\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:21.083524 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:21.083519 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:34:21.083926 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:21.083605 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3bbd20c-73f5-4d24-aed9-145e4b89a590-metrics-certs podName:e3bbd20c-73f5-4d24-aed9-145e4b89a590 nodeName:}" failed. No retries permitted until 2026-07-07 14:34:21.583579291 +0000 UTC m=+3.007765768 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3bbd20c-73f5-4d24-aed9-145e4b89a590-metrics-certs") pod "network-metrics-daemon-lpmtl" (UID: "e3bbd20c-73f5-4d24-aed9-145e4b89a590") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:34:21.083926 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.083660 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/89159896-e9f7-42e0-93e7-23122a296d8c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9t7xp\" (UID: \"89159896-e9f7-42e0-93e7-23122a296d8c\") " pod="openshift-multus/multus-additional-cni-plugins-9t7xp" Jul 07 14:34:21.083926 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.083721 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-host-run-netns\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:21.083926 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.083766 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/303d9d00-aaf9-431e-9980-60fd60904fad-host-run-netns\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:21.083926 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:21.083872 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 14:34:21.083926 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.083419 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/303d9d00-aaf9-431e-9980-60fd60904fad-var-lib-openvswitch\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:21.083926 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.083916 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/303d9d00-aaf9-431e-9980-60fd60904fad-run-ovn\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:21.083926 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:21.083926 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/686ecb3c-7c9a-4ac1-ac27-4430651b0870-original-pull-secret podName:686ecb3c-7c9a-4ac1-ac27-4430651b0870 nodeName:}" failed. No retries permitted until 2026-07-07 14:34:21.583908132 +0000 UTC m=+3.008094598 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/686ecb3c-7c9a-4ac1-ac27-4430651b0870-original-pull-secret") pod "global-pull-secret-syncer-bq6hn" (UID: "686ecb3c-7c9a-4ac1-ac27-4430651b0870") : object "kube-system"/"original-pull-secret" not registered Jul 07 14:34:21.084237 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.083952 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/303d9d00-aaf9-431e-9980-60fd60904fad-run-ovn\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:21.084237 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.083962 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/89159896-e9f7-42e0-93e7-23122a296d8c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9t7xp\" (UID: \"89159896-e9f7-42e0-93e7-23122a296d8c\") " pod="openshift-multus/multus-additional-cni-plugins-9t7xp" Jul 07 14:34:21.084237 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.083991 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c06aa1d3-1eb1-4a64-9677-2b7328a9cff9-iptables-alerter-script\") pod \"iptables-alerter-mj6xk\" (UID: \"c06aa1d3-1eb1-4a64-9677-2b7328a9cff9\") " pod="openshift-network-operator/iptables-alerter-mj6xk" Jul 07 14:34:21.084237 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.083995 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmzxg\" (UniqueName: \"kubernetes.io/projected/ff6cf858-eb9e-4adc-b4ae-9671841ef488-kube-api-access-lmzxg\") pod \"aws-ebs-csi-driver-node-sb4h5\" (UID: \"ff6cf858-eb9e-4adc-b4ae-9671841ef488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb4h5" Jul 07 14:34:21.084237 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.084047 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/303d9d00-aaf9-431e-9980-60fd60904fad-var-lib-openvswitch\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:21.084237 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.084027 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-host-run-multus-certs\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:21.084237 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.084093 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c06aa1d3-1eb1-4a64-9677-2b7328a9cff9-host-slash\") pod \"iptables-alerter-mj6xk\" (UID: \"c06aa1d3-1eb1-4a64-9677-2b7328a9cff9\") " pod="openshift-network-operator/iptables-alerter-mj6xk" Jul 07 14:34:21.084237 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.084097 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7d94747d-5e8a-4eab-8234-969f7dfbaee5-host\") pod \"node-ca-vkwws\" (UID: \"7d94747d-5e8a-4eab-8234-969f7dfbaee5\") " pod="openshift-image-registry/node-ca-vkwws" Jul 07 14:34:21.084237 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.084127 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/303d9d00-aaf9-431e-9980-60fd60904fad-env-overrides\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:21.084237 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.084133 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/303d9d00-aaf9-431e-9980-60fd60904fad-host-slash\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:21.084237 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.084155 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ff6cf858-eb9e-4adc-b4ae-9671841ef488-etc-selinux\") pod \"aws-ebs-csi-driver-node-sb4h5\" (UID: \"ff6cf858-eb9e-4adc-b4ae-9671841ef488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb4h5" Jul 07 14:34:21.084237 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.084179 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/303d9d00-aaf9-431e-9980-60fd60904fad-etc-openvswitch\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:21.084237 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.084181 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-cni-binary-copy\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:21.084237 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.084219 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/303d9d00-aaf9-431e-9980-60fd60904fad-ovnkube-script-lib\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:21.084844 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.084247 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmb7b\" (UniqueName: \"kubernetes.io/projected/89159896-e9f7-42e0-93e7-23122a296d8c-kube-api-access-dmb7b\") pod \"multus-additional-cni-plugins-9t7xp\" (UID: \"89159896-e9f7-42e0-93e7-23122a296d8c\") " pod="openshift-multus/multus-additional-cni-plugins-9t7xp" Jul 07 14:34:21.084844 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.084277 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-host-run-k8s-cni-cncf-io\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:21.084844 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.084303 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/686ecb3c-7c9a-4ac1-ac27-4430651b0870-dbus\") pod \"global-pull-secret-syncer-bq6hn\" (UID: \"686ecb3c-7c9a-4ac1-ac27-4430651b0870\") " pod="kube-system/global-pull-secret-syncer-bq6hn" Jul 07 14:34:21.084844 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.084331 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8lcx\" (UniqueName: \"kubernetes.io/projected/3e0c7b26-181b-47f2-8503-b66092a86230-kube-api-access-r8lcx\") pod \"network-check-target-zsq7v\" (UID: \"3e0c7b26-181b-47f2-8503-b66092a86230\") " pod="openshift-network-diagnostics/network-check-target-zsq7v" Jul 07 14:34:21.084844 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.084362 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/303d9d00-aaf9-431e-9980-60fd60904fad-host-run-ovn-kubernetes\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:21.084844 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.084389 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/303d9d00-aaf9-431e-9980-60fd60904fad-host-kubelet\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:21.084844 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.084415 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/303d9d00-aaf9-431e-9980-60fd60904fad-run-systemd\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:21.084844 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.084441 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/303d9d00-aaf9-431e-9980-60fd60904fad-run-openvswitch\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:21.084844 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.084468 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/303d9d00-aaf9-431e-9980-60fd60904fad-host-cni-netd\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:21.084844 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.084493 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ff6cf858-eb9e-4adc-b4ae-9671841ef488-sys-fs\") pod \"aws-ebs-csi-driver-node-sb4h5\" (UID: \"ff6cf858-eb9e-4adc-b4ae-9671841ef488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb4h5" Jul 07 14:34:21.084844 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.084524 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89159896-e9f7-42e0-93e7-23122a296d8c-system-cni-dir\") pod \"multus-additional-cni-plugins-9t7xp\" (UID: \"89159896-e9f7-42e0-93e7-23122a296d8c\") " pod="openshift-multus/multus-additional-cni-plugins-9t7xp" Jul 07 14:34:21.084844 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.084561 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89159896-e9f7-42e0-93e7-23122a296d8c-cnibin\") pod \"multus-additional-cni-plugins-9t7xp\" (UID: \"89159896-e9f7-42e0-93e7-23122a296d8c\") " pod="openshift-multus/multus-additional-cni-plugins-9t7xp" Jul 07 14:34:21.084844 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.084590 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/89159896-e9f7-42e0-93e7-23122a296d8c-os-release\") pod \"multus-additional-cni-plugins-9t7xp\" (UID: \"89159896-e9f7-42e0-93e7-23122a296d8c\") " pod="openshift-multus/multus-additional-cni-plugins-9t7xp" Jul 07 14:34:21.084844 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.084618 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9ptv\" (UniqueName: \"kubernetes.io/projected/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-kube-api-access-k9ptv\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:21.084844 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.084644 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7d94747d-5e8a-4eab-8234-969f7dfbaee5-serviceca\") pod \"node-ca-vkwws\" (UID: \"7d94747d-5e8a-4eab-8234-969f7dfbaee5\") " pod="openshift-image-registry/node-ca-vkwws" Jul 07 14:34:21.084844 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.084689 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/303d9d00-aaf9-431e-9980-60fd60904fad-log-socket\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:21.084844 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.084705 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-cni-binary-copy\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:21.085676 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.084721 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/303d9d00-aaf9-431e-9980-60fd60904fad-host-cni-bin\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:21.085676 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.084746 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/553f749a-649c-41c0-aa39-a0cb13725ca8-tmp-dir\") pod \"node-resolver-czppd\" (UID: \"553f749a-649c-41c0-aa39-a0cb13725ca8\") " pod="openshift-dns/node-resolver-czppd" Jul 07 14:34:21.085676 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.084763 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/303d9d00-aaf9-431e-9980-60fd60904fad-systemd-units\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:21.085676 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.084772 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89159896-e9f7-42e0-93e7-23122a296d8c-cni-binary-copy\") pod \"multus-additional-cni-plugins-9t7xp\" (UID: \"89159896-e9f7-42e0-93e7-23122a296d8c\") " pod="openshift-multus/multus-additional-cni-plugins-9t7xp" Jul 07 14:34:21.085676 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.084800 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-host-var-lib-cni-multus\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:21.085676 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.084847 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-host-var-lib-kubelet\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:21.085676 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.084884 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-multus-daemon-config\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:21.085676 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.084929 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ff6cf858-eb9e-4adc-b4ae-9671841ef488-socket-dir\") pod \"aws-ebs-csi-driver-node-sb4h5\" (UID: \"ff6cf858-eb9e-4adc-b4ae-9671841ef488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb4h5" Jul 07 14:34:21.085676 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.084958 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/303d9d00-aaf9-431e-9980-60fd60904fad-node-log\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:21.085676 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.084984 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/553f749a-649c-41c0-aa39-a0cb13725ca8-hosts-file\") pod \"node-resolver-czppd\" (UID: \"553f749a-649c-41c0-aa39-a0cb13725ca8\") " pod="openshift-dns/node-resolver-czppd" Jul 07 14:34:21.085676 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.085012 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g29pn\" (UniqueName: \"kubernetes.io/projected/553f749a-649c-41c0-aa39-a0cb13725ca8-kube-api-access-g29pn\") pod \"node-resolver-czppd\" (UID: \"553f749a-649c-41c0-aa39-a0cb13725ca8\") " pod="openshift-dns/node-resolver-czppd" Jul 07 14:34:21.085676 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.085039 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ff6cf858-eb9e-4adc-b4ae-9671841ef488-registration-dir\") pod \"aws-ebs-csi-driver-node-sb4h5\" (UID: \"ff6cf858-eb9e-4adc-b4ae-9671841ef488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb4h5" Jul 07 14:34:21.085676 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.085050 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-host-run-multus-certs\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:21.085676 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.085065 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ff6cf858-eb9e-4adc-b4ae-9671841ef488-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sb4h5\" (UID: \"ff6cf858-eb9e-4adc-b4ae-9671841ef488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb4h5" Jul 07 14:34:21.085676 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.085091 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-os-release\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:21.085676 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.085102 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7d94747d-5e8a-4eab-8234-969f7dfbaee5-host\") pod \"node-ca-vkwws\" (UID: \"7d94747d-5e8a-4eab-8234-969f7dfbaee5\") " pod="openshift-image-registry/node-ca-vkwws" Jul 07 14:34:21.085676 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.085117 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-host-var-lib-cni-bin\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:21.086268 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.085148 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/686ecb3c-7c9a-4ac1-ac27-4430651b0870-kubelet-config\") pod \"global-pull-secret-syncer-bq6hn\" (UID: \"686ecb3c-7c9a-4ac1-ac27-4430651b0870\") " pod="kube-system/global-pull-secret-syncer-bq6hn" Jul 07 14:34:21.086268 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.085176 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/303d9d00-aaf9-431e-9980-60fd60904fad-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:21.086268 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.085203 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89159896-e9f7-42e0-93e7-23122a296d8c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9t7xp\" (UID: \"89159896-e9f7-42e0-93e7-23122a296d8c\") " pod="openshift-multus/multus-additional-cni-plugins-9t7xp" Jul 07 14:34:21.086268 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.085230 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ff6cf858-eb9e-4adc-b4ae-9671841ef488-device-dir\") pod \"aws-ebs-csi-driver-node-sb4h5\" (UID: \"ff6cf858-eb9e-4adc-b4ae-9671841ef488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb4h5" Jul 07 14:34:21.086268 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.085257 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-multus-socket-dir-parent\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:21.086268 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.085286 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/303d9d00-aaf9-431e-9980-60fd60904fad-ovnkube-config\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:21.086268 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.085316 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8pp8\" (UniqueName: \"kubernetes.io/projected/e3bbd20c-73f5-4d24-aed9-145e4b89a590-kube-api-access-w8pp8\") pod \"network-metrics-daemon-lpmtl\" (UID: \"e3bbd20c-73f5-4d24-aed9-145e4b89a590\") " pod="openshift-multus/network-metrics-daemon-lpmtl" Jul 07 14:34:21.086268 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.085344 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-system-cni-dir\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:21.086268 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.085372 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-etc-kubernetes\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:21.086268 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.085466 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-etc-kubernetes\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:21.086268 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.085531 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/303d9d00-aaf9-431e-9980-60fd60904fad-env-overrides\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:21.086268 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.085644 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ff6cf858-eb9e-4adc-b4ae-9671841ef488-etc-selinux\") pod \"aws-ebs-csi-driver-node-sb4h5\" (UID: \"ff6cf858-eb9e-4adc-b4ae-9671841ef488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb4h5" Jul 07 14:34:21.086268 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.085700 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/303d9d00-aaf9-431e-9980-60fd60904fad-run-systemd\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:21.086268 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.085793 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/553f749a-649c-41c0-aa39-a0cb13725ca8-tmp-dir\") pod \"node-resolver-czppd\" (UID: \"553f749a-649c-41c0-aa39-a0cb13725ca8\") " pod="openshift-dns/node-resolver-czppd" Jul 07 14:34:21.086268 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.085870 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/303d9d00-aaf9-431e-9980-60fd60904fad-run-openvswitch\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:21.086268 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.085879 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-multus-cni-dir\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:21.086268 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.085938 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ff6cf858-eb9e-4adc-b4ae-9671841ef488-sys-fs\") pod \"aws-ebs-csi-driver-node-sb4h5\" (UID: \"ff6cf858-eb9e-4adc-b4ae-9671841ef488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb4h5" Jul 07 14:34:21.087149 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.085928 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/303d9d00-aaf9-431e-9980-60fd60904fad-host-cni-netd\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:21.087149 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.086003 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89159896-e9f7-42e0-93e7-23122a296d8c-system-cni-dir\") pod \"multus-additional-cni-plugins-9t7xp\" (UID: \"89159896-e9f7-42e0-93e7-23122a296d8c\") " pod="openshift-multus/multus-additional-cni-plugins-9t7xp" Jul 07 14:34:21.087149 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.086048 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89159896-e9f7-42e0-93e7-23122a296d8c-cnibin\") pod \"multus-additional-cni-plugins-9t7xp\" (UID: \"89159896-e9f7-42e0-93e7-23122a296d8c\") " pod="openshift-multus/multus-additional-cni-plugins-9t7xp" Jul 07 14:34:21.087149 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.086110 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/89159896-e9f7-42e0-93e7-23122a296d8c-os-release\") pod \"multus-additional-cni-plugins-9t7xp\" (UID: \"89159896-e9f7-42e0-93e7-23122a296d8c\") " pod="openshift-multus/multus-additional-cni-plugins-9t7xp" Jul 07 14:34:21.087149 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.086182 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89159896-e9f7-42e0-93e7-23122a296d8c-cni-binary-copy\") pod \"multus-additional-cni-plugins-9t7xp\" (UID: \"89159896-e9f7-42e0-93e7-23122a296d8c\") " pod="openshift-multus/multus-additional-cni-plugins-9t7xp" Jul 07 14:34:21.087149 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.086274 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-os-release\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:21.087149 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.086344 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-multus-conf-dir\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:21.087149 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.086390 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-host-var-lib-cni-multus\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:21.087149 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.086429 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-host-var-lib-kubelet\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:21.087149 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.086425 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/89159896-e9f7-42e0-93e7-23122a296d8c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9t7xp\" (UID: \"89159896-e9f7-42e0-93e7-23122a296d8c\") " pod="openshift-multus/multus-additional-cni-plugins-9t7xp" Jul 07 14:34:21.087149 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.086479 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-host-var-lib-cni-bin\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:21.087149 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.086493 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-host-run-k8s-cni-cncf-io\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:21.087149 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.086557 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/303d9d00-aaf9-431e-9980-60fd60904fad-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:21.087149 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.086561 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/686ecb3c-7c9a-4ac1-ac27-4430651b0870-dbus\") pod \"global-pull-secret-syncer-bq6hn\" (UID: \"686ecb3c-7c9a-4ac1-ac27-4430651b0870\") " pod="kube-system/global-pull-secret-syncer-bq6hn" Jul 07 14:34:21.087149 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.086597 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/686ecb3c-7c9a-4ac1-ac27-4430651b0870-kubelet-config\") pod \"global-pull-secret-syncer-bq6hn\" (UID: \"686ecb3c-7c9a-4ac1-ac27-4430651b0870\") " pod="kube-system/global-pull-secret-syncer-bq6hn" Jul 07 14:34:21.087149 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.086634 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ff6cf858-eb9e-4adc-b4ae-9671841ef488-device-dir\") pod \"aws-ebs-csi-driver-node-sb4h5\" (UID: \"ff6cf858-eb9e-4adc-b4ae-9671841ef488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb4h5" Jul 07 14:34:21.087149 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.086694 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ff6cf858-eb9e-4adc-b4ae-9671841ef488-registration-dir\") pod \"aws-ebs-csi-driver-node-sb4h5\" (UID: \"ff6cf858-eb9e-4adc-b4ae-9671841ef488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb4h5" Jul 07 14:34:21.087809 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.086704 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-multus-socket-dir-parent\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:21.087809 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.086718 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7d94747d-5e8a-4eab-8234-969f7dfbaee5-serviceca\") pod \"node-ca-vkwws\" (UID: \"7d94747d-5e8a-4eab-8234-969f7dfbaee5\") " pod="openshift-image-registry/node-ca-vkwws" Jul 07 14:34:21.087809 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.086715 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/303d9d00-aaf9-431e-9980-60fd60904fad-host-cni-bin\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:21.087809 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.086735 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/303d9d00-aaf9-431e-9980-60fd60904fad-host-kubelet\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:21.087809 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.086876 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/303d9d00-aaf9-431e-9980-60fd60904fad-host-run-ovn-kubernetes\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:21.087809 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.086925 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-system-cni-dir\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:21.087809 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.087023 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/303d9d00-aaf9-431e-9980-60fd60904fad-node-log\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:21.087809 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.087028 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ff6cf858-eb9e-4adc-b4ae-9671841ef488-socket-dir\") pod \"aws-ebs-csi-driver-node-sb4h5\" (UID: \"ff6cf858-eb9e-4adc-b4ae-9671841ef488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb4h5" Jul 07 14:34:21.087809 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.087073 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/553f749a-649c-41c0-aa39-a0cb13725ca8-hosts-file\") pod \"node-resolver-czppd\" (UID: \"553f749a-649c-41c0-aa39-a0cb13725ca8\") " pod="openshift-dns/node-resolver-czppd" Jul 07 14:34:21.087809 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.087124 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ff6cf858-eb9e-4adc-b4ae-9671841ef488-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sb4h5\" (UID: \"ff6cf858-eb9e-4adc-b4ae-9671841ef488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb4h5" Jul 07 14:34:21.087809 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.087169 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/303d9d00-aaf9-431e-9980-60fd60904fad-log-socket\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:21.087809 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.087245 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/303d9d00-aaf9-431e-9980-60fd60904fad-ovnkube-config\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:21.087809 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.087436 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89159896-e9f7-42e0-93e7-23122a296d8c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9t7xp\" (UID: \"89159896-e9f7-42e0-93e7-23122a296d8c\") " pod="openshift-multus/multus-additional-cni-plugins-9t7xp" Jul 07 14:34:21.087809 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.087447 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-multus-daemon-config\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:21.087809 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.087668 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/303d9d00-aaf9-431e-9980-60fd60904fad-ovnkube-script-lib\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:21.089238 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.089211 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/303d9d00-aaf9-431e-9980-60fd60904fad-ovn-node-metrics-cert\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:21.094030 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.093647 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgnsk\" (UniqueName: \"kubernetes.io/projected/c06aa1d3-1eb1-4a64-9677-2b7328a9cff9-kube-api-access-mgnsk\") pod \"iptables-alerter-mj6xk\" (UID: \"c06aa1d3-1eb1-4a64-9677-2b7328a9cff9\") " pod="openshift-network-operator/iptables-alerter-mj6xk" Jul 07 14:34:21.094365 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:21.094339 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 14:34:21.094485 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:21.094371 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 14:34:21.094485 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:21.094392 2555 projected.go:196] Error preparing data for projected volume kube-api-access-r8lcx for pod openshift-network-diagnostics/network-check-target-zsq7v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:34:21.094485 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:21.094461 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3e0c7b26-181b-47f2-8503-b66092a86230-kube-api-access-r8lcx podName:3e0c7b26-181b-47f2-8503-b66092a86230 nodeName:}" failed. No retries permitted until 2026-07-07 14:34:21.594444151 +0000 UTC m=+3.018630631 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-r8lcx" (UniqueName: "kubernetes.io/projected/3e0c7b26-181b-47f2-8503-b66092a86230-kube-api-access-r8lcx") pod "network-check-target-zsq7v" (UID: "3e0c7b26-181b-47f2-8503-b66092a86230") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:34:21.095741 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.095569 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9ptv\" (UniqueName: \"kubernetes.io/projected/f5f5ff03-a3bc-4758-8023-e023ab4ed3b3-kube-api-access-k9ptv\") pod \"multus-bfcr2\" (UID: \"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3\") " pod="openshift-multus/multus-bfcr2" Jul 07 14:34:21.095741 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.095692 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmzxg\" (UniqueName: \"kubernetes.io/projected/ff6cf858-eb9e-4adc-b4ae-9671841ef488-kube-api-access-lmzxg\") pod \"aws-ebs-csi-driver-node-sb4h5\" (UID: \"ff6cf858-eb9e-4adc-b4ae-9671841ef488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb4h5" Jul 07 14:34:21.096972 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.096915 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8srjl\" (UniqueName: \"kubernetes.io/projected/303d9d00-aaf9-431e-9980-60fd60904fad-kube-api-access-8srjl\") pod \"ovnkube-node-59hn4\" (UID: \"303d9d00-aaf9-431e-9980-60fd60904fad\") " pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:21.097695 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.097396 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8pp8\" (UniqueName: \"kubernetes.io/projected/e3bbd20c-73f5-4d24-aed9-145e4b89a590-kube-api-access-w8pp8\") pod \"network-metrics-daemon-lpmtl\" (UID: \"e3bbd20c-73f5-4d24-aed9-145e4b89a590\") " pod="openshift-multus/network-metrics-daemon-lpmtl" Jul 07 14:34:21.097695 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.097668 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmb7b\" (UniqueName: \"kubernetes.io/projected/89159896-e9f7-42e0-93e7-23122a296d8c-kube-api-access-dmb7b\") pod \"multus-additional-cni-plugins-9t7xp\" (UID: \"89159896-e9f7-42e0-93e7-23122a296d8c\") " pod="openshift-multus/multus-additional-cni-plugins-9t7xp" Jul 07 14:34:21.098392 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.098253 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlmm6\" (UniqueName: \"kubernetes.io/projected/7d94747d-5e8a-4eab-8234-969f7dfbaee5-kube-api-access-qlmm6\") pod \"node-ca-vkwws\" (UID: \"7d94747d-5e8a-4eab-8234-969f7dfbaee5\") " pod="openshift-image-registry/node-ca-vkwws" Jul 07 14:34:21.098503 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.098405 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g29pn\" (UniqueName: \"kubernetes.io/projected/553f749a-649c-41c0-aa39-a0cb13725ca8-kube-api-access-g29pn\") pod \"node-resolver-czppd\" (UID: \"553f749a-649c-41c0-aa39-a0cb13725ca8\") " pod="openshift-dns/node-resolver-czppd" Jul 07 14:34:21.182799 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.182744 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4zclx" Jul 07 14:34:21.192927 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.192897 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" Jul 07 14:34:21.202982 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.202953 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9t7xp" Jul 07 14:34:21.212076 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.212054 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-czppd" Jul 07 14:34:21.220016 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.219992 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb4h5" Jul 07 14:34:21.227959 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.227939 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bfcr2" Jul 07 14:34:21.237863 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.237833 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mj6xk" Jul 07 14:34:21.246624 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.246603 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:21.252583 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.252560 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vkwws" Jul 07 14:34:21.588992 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.588948 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/686ecb3c-7c9a-4ac1-ac27-4430651b0870-original-pull-secret\") pod \"global-pull-secret-syncer-bq6hn\" (UID: \"686ecb3c-7c9a-4ac1-ac27-4430651b0870\") " pod="kube-system/global-pull-secret-syncer-bq6hn" Jul 07 14:34:21.589194 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.589003 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3bbd20c-73f5-4d24-aed9-145e4b89a590-metrics-certs\") pod \"network-metrics-daemon-lpmtl\" (UID: \"e3bbd20c-73f5-4d24-aed9-145e4b89a590\") " pod="openshift-multus/network-metrics-daemon-lpmtl" Jul 07 14:34:21.589194 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:21.589135 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 14:34:21.589363 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:21.589228 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/686ecb3c-7c9a-4ac1-ac27-4430651b0870-original-pull-secret podName:686ecb3c-7c9a-4ac1-ac27-4430651b0870 nodeName:}" failed. No retries permitted until 2026-07-07 14:34:22.58920415 +0000 UTC m=+4.013390621 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/686ecb3c-7c9a-4ac1-ac27-4430651b0870-original-pull-secret") pod "global-pull-secret-syncer-bq6hn" (UID: "686ecb3c-7c9a-4ac1-ac27-4430651b0870") : object "kube-system"/"original-pull-secret" not registered Jul 07 14:34:21.589363 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:21.589135 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:34:21.589363 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:21.589324 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3bbd20c-73f5-4d24-aed9-145e4b89a590-metrics-certs podName:e3bbd20c-73f5-4d24-aed9-145e4b89a590 nodeName:}" failed. No retries permitted until 2026-07-07 14:34:22.589300009 +0000 UTC m=+4.013486469 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3bbd20c-73f5-4d24-aed9-145e4b89a590-metrics-certs") pod "network-metrics-daemon-lpmtl" (UID: "e3bbd20c-73f5-4d24-aed9-145e4b89a590") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:34:21.690221 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.690174 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8lcx\" (UniqueName: \"kubernetes.io/projected/3e0c7b26-181b-47f2-8503-b66092a86230-kube-api-access-r8lcx\") pod \"network-check-target-zsq7v\" (UID: \"3e0c7b26-181b-47f2-8503-b66092a86230\") " pod="openshift-network-diagnostics/network-check-target-zsq7v" Jul 07 14:34:21.690418 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:21.690365 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 14:34:21.690418 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:21.690392 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 14:34:21.690418 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:21.690408 2555 projected.go:196] Error preparing data for projected volume kube-api-access-r8lcx for pod openshift-network-diagnostics/network-check-target-zsq7v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:34:21.690569 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:21.690479 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3e0c7b26-181b-47f2-8503-b66092a86230-kube-api-access-r8lcx podName:3e0c7b26-181b-47f2-8503-b66092a86230 nodeName:}" failed. No retries permitted until 2026-07-07 14:34:22.690459015 +0000 UTC m=+4.114645488 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-r8lcx" (UniqueName: "kubernetes.io/projected/3e0c7b26-181b-47f2-8503-b66092a86230-kube-api-access-r8lcx") pod "network-check-target-zsq7v" (UID: "3e0c7b26-181b-47f2-8503-b66092a86230") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:34:21.921336 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.921242 2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 14:29:21 +0000 UTC" deadline="2028-04-10 08:56:42.498857424 +0000 UTC" Jul 07 14:34:21.921336 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:21.921284 2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15426h22m20.57758288s" Jul 07 14:34:22.036468 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:22.036433 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpmtl" Jul 07 14:34:22.036622 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:22.036433 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zsq7v" Jul 07 14:34:22.036622 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:22.036568 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpmtl" podUID="e3bbd20c-73f5-4d24-aed9-145e4b89a590" Jul 07 14:34:22.036710 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:22.036433 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bq6hn" Jul 07 14:34:22.036755 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:22.036631 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zsq7v" podUID="3e0c7b26-181b-47f2-8503-b66092a86230" Jul 07 14:34:22.036815 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:22.036798 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bq6hn" podUID="686ecb3c-7c9a-4ac1-ac27-4430651b0870" Jul 07 14:34:22.155175 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:22.155136 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc06aa1d3_1eb1_4a64_9677_2b7328a9cff9.slice/crio-5185c9a13f4b0a2e53a7d158ffc056ea4ee8af581f3db15358e1a4716ec8d766 WatchSource:0}: Error finding container 5185c9a13f4b0a2e53a7d158ffc056ea4ee8af581f3db15358e1a4716ec8d766: Status 404 returned error can't find the container with id 5185c9a13f4b0a2e53a7d158ffc056ea4ee8af581f3db15358e1a4716ec8d766 Jul 07 14:34:22.158380 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:22.158339 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5f5ff03_a3bc_4758_8023_e023ab4ed3b3.slice/crio-ce9f76deb3eb6a1fbbddee5a1522beb154bec9b8cb5fbe135de2ef49f8856942 WatchSource:0}: Error finding container ce9f76deb3eb6a1fbbddee5a1522beb154bec9b8cb5fbe135de2ef49f8856942: Status 404 returned error can't find the container with id ce9f76deb3eb6a1fbbddee5a1522beb154bec9b8cb5fbe135de2ef49f8856942 Jul 07 14:34:22.161627 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:22.161597 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff6cf858_eb9e_4adc_b4ae_9671841ef488.slice/crio-40846fca7942192d5a3dc590a909af9842ff50b4093c027bfa9dcb83fea41217 WatchSource:0}: Error finding container 40846fca7942192d5a3dc590a909af9842ff50b4093c027bfa9dcb83fea41217: Status 404 returned error can't find the container with id 40846fca7942192d5a3dc590a909af9842ff50b4093c027bfa9dcb83fea41217 Jul 07 14:34:22.162656 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:22.162630 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d94747d_5e8a_4eab_8234_969f7dfbaee5.slice/crio-b26a1e442f5e4573b8d981693d65499d48f861019babfffe7adfd6ef475f6646 WatchSource:0}: Error finding container b26a1e442f5e4573b8d981693d65499d48f861019babfffe7adfd6ef475f6646: Status 404 returned error can't find the container with id b26a1e442f5e4573b8d981693d65499d48f861019babfffe7adfd6ef475f6646 Jul 07 14:34:22.163405 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:22.163289 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303d9d00_aaf9_431e_9980_60fd60904fad.slice/crio-08ae2d2c23e1b285763fa6f5fec73b98aa2f341499101e2b5cd528d410f28861 WatchSource:0}: Error finding container 08ae2d2c23e1b285763fa6f5fec73b98aa2f341499101e2b5cd528d410f28861: Status 404 returned error can't find the container with id 08ae2d2c23e1b285763fa6f5fec73b98aa2f341499101e2b5cd528d410f28861 Jul 07 14:34:22.164373 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:22.164056 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89159896_e9f7_42e0_93e7_23122a296d8c.slice/crio-d45cddc9550a8e18e5a466c0ca256e8aded35dd73e969021ebf6594394d26378 WatchSource:0}: Error finding container d45cddc9550a8e18e5a466c0ca256e8aded35dd73e969021ebf6594394d26378: Status 404 returned error can't find the container with id d45cddc9550a8e18e5a466c0ca256e8aded35dd73e969021ebf6594394d26378 Jul 07 14:34:22.166935 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:22.166406 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaf88254_ff84_430f_b609_debc1b4b170c.slice/crio-09b5af1ea99ca76e0d55c45bc3f2e24fbccea5d5e7b80204a9fd55f73bb99dc4 WatchSource:0}: Error finding container 09b5af1ea99ca76e0d55c45bc3f2e24fbccea5d5e7b80204a9fd55f73bb99dc4: Status 404 returned error can't find the container with id 09b5af1ea99ca76e0d55c45bc3f2e24fbccea5d5e7b80204a9fd55f73bb99dc4 Jul 07 14:34:22.168108 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:22.167605 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod553f749a_649c_41c0_aa39_a0cb13725ca8.slice/crio-b075eb729504c82c93ca8c7684c00fd4f02dbf03f55acfaba96d90019f01dd8e WatchSource:0}: Error finding container b075eb729504c82c93ca8c7684c00fd4f02dbf03f55acfaba96d90019f01dd8e: Status 404 returned error can't find the container with id b075eb729504c82c93ca8c7684c00fd4f02dbf03f55acfaba96d90019f01dd8e Jul 07 14:34:22.168591 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:22.168459 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ea25b41_29d9_4e81_b4d5_6252d76a8e66.slice/crio-d67f54b9d7f127911464c8833c2ab8f762f4840d37719bd3f4c90ce4037fc730 WatchSource:0}: Error finding container d67f54b9d7f127911464c8833c2ab8f762f4840d37719bd3f4c90ce4037fc730: Status 404 returned error can't find the container with id d67f54b9d7f127911464c8833c2ab8f762f4840d37719bd3f4c90ce4037fc730 Jul 07 14:34:22.598801 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:22.598526 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/686ecb3c-7c9a-4ac1-ac27-4430651b0870-original-pull-secret\") pod \"global-pull-secret-syncer-bq6hn\" (UID: \"686ecb3c-7c9a-4ac1-ac27-4430651b0870\") " pod="kube-system/global-pull-secret-syncer-bq6hn" Jul 07 14:34:22.598986 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:22.598852 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3bbd20c-73f5-4d24-aed9-145e4b89a590-metrics-certs\") pod \"network-metrics-daemon-lpmtl\" (UID: \"e3bbd20c-73f5-4d24-aed9-145e4b89a590\") " pod="openshift-multus/network-metrics-daemon-lpmtl" Jul 07 14:34:22.598986 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:22.598728 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 14:34:22.599125 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:22.598989 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/686ecb3c-7c9a-4ac1-ac27-4430651b0870-original-pull-secret podName:686ecb3c-7c9a-4ac1-ac27-4430651b0870 nodeName:}" failed. No retries permitted until 2026-07-07 14:34:24.598960121 +0000 UTC m=+6.023146600 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/686ecb3c-7c9a-4ac1-ac27-4430651b0870-original-pull-secret") pod "global-pull-secret-syncer-bq6hn" (UID: "686ecb3c-7c9a-4ac1-ac27-4430651b0870") : object "kube-system"/"original-pull-secret" not registered Jul 07 14:34:22.599125 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:22.599027 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:34:22.599125 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:22.599106 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3bbd20c-73f5-4d24-aed9-145e4b89a590-metrics-certs podName:e3bbd20c-73f5-4d24-aed9-145e4b89a590 nodeName:}" failed. No retries permitted until 2026-07-07 14:34:24.599088739 +0000 UTC m=+6.023275197 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3bbd20c-73f5-4d24-aed9-145e4b89a590-metrics-certs") pod "network-metrics-daemon-lpmtl" (UID: "e3bbd20c-73f5-4d24-aed9-145e4b89a590") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:34:22.700713 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:22.699927 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8lcx\" (UniqueName: \"kubernetes.io/projected/3e0c7b26-181b-47f2-8503-b66092a86230-kube-api-access-r8lcx\") pod \"network-check-target-zsq7v\" (UID: \"3e0c7b26-181b-47f2-8503-b66092a86230\") " pod="openshift-network-diagnostics/network-check-target-zsq7v" Jul 07 14:34:22.700713 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:22.700146 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 14:34:22.700713 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:22.700180 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 14:34:22.700713 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:22.700195 2555 projected.go:196] Error preparing data for projected volume kube-api-access-r8lcx for pod openshift-network-diagnostics/network-check-target-zsq7v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:34:22.700713 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:22.700303 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3e0c7b26-181b-47f2-8503-b66092a86230-kube-api-access-r8lcx podName:3e0c7b26-181b-47f2-8503-b66092a86230 nodeName:}" failed. No retries permitted until 2026-07-07 14:34:24.700281097 +0000 UTC m=+6.124467567 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-r8lcx" (UniqueName: "kubernetes.io/projected/3e0c7b26-181b-47f2-8503-b66092a86230-kube-api-access-r8lcx") pod "network-check-target-zsq7v" (UID: "3e0c7b26-181b-47f2-8503-b66092a86230") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:34:22.922488 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:22.922417 2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 14:29:21 +0000 UTC" deadline="2028-04-19 14:51:40.65138923 +0000 UTC" Jul 07 14:34:22.922488 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:22.922486 2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15648h17m17.728908237s" Jul 07 14:34:23.058867 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:23.058801 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4zclx" event={"ID":"3ea25b41-29d9-4e81-b4d5-6252d76a8e66","Type":"ContainerStarted","Data":"d67f54b9d7f127911464c8833c2ab8f762f4840d37719bd3f4c90ce4037fc730"} Jul 07 14:34:23.062933 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:23.062898 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vkwws" event={"ID":"7d94747d-5e8a-4eab-8234-969f7dfbaee5","Type":"ContainerStarted","Data":"b26a1e442f5e4573b8d981693d65499d48f861019babfffe7adfd6ef475f6646"} Jul 07 14:34:23.065562 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:23.065532 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-czppd" event={"ID":"553f749a-649c-41c0-aa39-a0cb13725ca8","Type":"ContainerStarted","Data":"b075eb729504c82c93ca8c7684c00fd4f02dbf03f55acfaba96d90019f01dd8e"} Jul 07 14:34:23.068311 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:23.068280 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" event={"ID":"aaf88254-ff84-430f-b609-debc1b4b170c","Type":"ContainerStarted","Data":"09b5af1ea99ca76e0d55c45bc3f2e24fbccea5d5e7b80204a9fd55f73bb99dc4"} Jul 07 14:34:23.070514 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:23.070481 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9t7xp" event={"ID":"89159896-e9f7-42e0-93e7-23122a296d8c","Type":"ContainerStarted","Data":"d45cddc9550a8e18e5a466c0ca256e8aded35dd73e969021ebf6594394d26378"} Jul 07 14:34:23.076931 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:23.076845 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb4h5" event={"ID":"ff6cf858-eb9e-4adc-b4ae-9671841ef488","Type":"ContainerStarted","Data":"40846fca7942192d5a3dc590a909af9842ff50b4093c027bfa9dcb83fea41217"} Jul 07 14:34:23.082656 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:23.081674 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-19.ec2.internal" event={"ID":"ff9a5e8afa26523332eda2af110e42ee","Type":"ContainerStarted","Data":"2e20bc2239f12af91712248cc03d07022d9fd100bc6e8d9858015d05a7f4f1ff"} Jul 07 14:34:23.090382 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:23.089758 2555 generic.go:356] "Generic (PLEG): container finished" podID="340cb4146c128f06bd339f0108cd15f2" containerID="081b5206381bcfb7b228fc8ce6aad9f6ba38866c19b369347126903971121953" exitCode=0 Jul 07 14:34:23.090665 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:23.090595 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-19.ec2.internal" event={"ID":"340cb4146c128f06bd339f0108cd15f2","Type":"ContainerDied","Data":"081b5206381bcfb7b228fc8ce6aad9f6ba38866c19b369347126903971121953"} Jul 07 14:34:23.101600 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:23.101473 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" event={"ID":"303d9d00-aaf9-431e-9980-60fd60904fad","Type":"ContainerStarted","Data":"08ae2d2c23e1b285763fa6f5fec73b98aa2f341499101e2b5cd528d410f28861"} Jul 07 14:34:23.109141 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:23.109071 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bfcr2" event={"ID":"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3","Type":"ContainerStarted","Data":"ce9f76deb3eb6a1fbbddee5a1522beb154bec9b8cb5fbe135de2ef49f8856942"} Jul 07 14:34:23.122237 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:23.122172 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-19.ec2.internal" podStartSLOduration=2.122151013 podStartE2EDuration="2.122151013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:34:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:34:23.101135402 +0000 UTC m=+4.525321883" watchObservedRunningTime="2026-07-07 14:34:23.122151013 +0000 UTC m=+4.546337497" Jul 07 14:34:23.125058 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:23.124989 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mj6xk" event={"ID":"c06aa1d3-1eb1-4a64-9677-2b7328a9cff9","Type":"ContainerStarted","Data":"5185c9a13f4b0a2e53a7d158ffc056ea4ee8af581f3db15358e1a4716ec8d766"} Jul 07 14:34:24.037006 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:24.036162 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bq6hn" Jul 07 14:34:24.037006 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:24.036306 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bq6hn" podUID="686ecb3c-7c9a-4ac1-ac27-4430651b0870" Jul 07 14:34:24.037006 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:24.036386 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zsq7v" Jul 07 14:34:24.037006 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:24.036455 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zsq7v" podUID="3e0c7b26-181b-47f2-8503-b66092a86230" Jul 07 14:34:24.037006 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:24.036516 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpmtl" Jul 07 14:34:24.037006 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:24.036600 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpmtl" podUID="e3bbd20c-73f5-4d24-aed9-145e4b89a590" Jul 07 14:34:24.140641 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:24.140602 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-19.ec2.internal" event={"ID":"340cb4146c128f06bd339f0108cd15f2","Type":"ContainerStarted","Data":"c560154592875461b57e95cf0a984cf1faa8c0be3be5d4c50de88149239f7fb1"} Jul 07 14:34:24.158598 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:24.157968 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-19.ec2.internal" podStartSLOduration=3.157943209 podStartE2EDuration="3.157943209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:34:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:34:24.157234219 +0000 UTC m=+5.581420699" watchObservedRunningTime="2026-07-07 14:34:24.157943209 +0000 UTC m=+5.582129689" Jul 07 14:34:24.619659 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:24.619026 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/686ecb3c-7c9a-4ac1-ac27-4430651b0870-original-pull-secret\") pod \"global-pull-secret-syncer-bq6hn\" (UID: \"686ecb3c-7c9a-4ac1-ac27-4430651b0870\") " pod="kube-system/global-pull-secret-syncer-bq6hn" Jul 07 14:34:24.619659 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:24.619085 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3bbd20c-73f5-4d24-aed9-145e4b89a590-metrics-certs\") pod \"network-metrics-daemon-lpmtl\" (UID: \"e3bbd20c-73f5-4d24-aed9-145e4b89a590\") " pod="openshift-multus/network-metrics-daemon-lpmtl" Jul 07 14:34:24.619659 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:24.619202 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 14:34:24.619659 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:24.619259 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:34:24.619659 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:24.619299 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/686ecb3c-7c9a-4ac1-ac27-4430651b0870-original-pull-secret podName:686ecb3c-7c9a-4ac1-ac27-4430651b0870 nodeName:}" failed. No retries permitted until 2026-07-07 14:34:28.619266606 +0000 UTC m=+10.043453068 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/686ecb3c-7c9a-4ac1-ac27-4430651b0870-original-pull-secret") pod "global-pull-secret-syncer-bq6hn" (UID: "686ecb3c-7c9a-4ac1-ac27-4430651b0870") : object "kube-system"/"original-pull-secret" not registered Jul 07 14:34:24.619659 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:24.619323 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3bbd20c-73f5-4d24-aed9-145e4b89a590-metrics-certs podName:e3bbd20c-73f5-4d24-aed9-145e4b89a590 nodeName:}" failed. No retries permitted until 2026-07-07 14:34:28.619310778 +0000 UTC m=+10.043497242 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3bbd20c-73f5-4d24-aed9-145e4b89a590-metrics-certs") pod "network-metrics-daemon-lpmtl" (UID: "e3bbd20c-73f5-4d24-aed9-145e4b89a590") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:34:24.719970 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:24.719904 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8lcx\" (UniqueName: \"kubernetes.io/projected/3e0c7b26-181b-47f2-8503-b66092a86230-kube-api-access-r8lcx\") pod \"network-check-target-zsq7v\" (UID: \"3e0c7b26-181b-47f2-8503-b66092a86230\") " pod="openshift-network-diagnostics/network-check-target-zsq7v" Jul 07 14:34:24.720171 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:24.720080 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 14:34:24.720171 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:24.720103 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 14:34:24.720171 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:24.720118 2555 projected.go:196] Error preparing data for projected volume kube-api-access-r8lcx for pod openshift-network-diagnostics/network-check-target-zsq7v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:34:24.720338 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:24.720190 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3e0c7b26-181b-47f2-8503-b66092a86230-kube-api-access-r8lcx podName:3e0c7b26-181b-47f2-8503-b66092a86230 nodeName:}" failed. No retries permitted until 2026-07-07 14:34:28.720167807 +0000 UTC m=+10.144354284 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-r8lcx" (UniqueName: "kubernetes.io/projected/3e0c7b26-181b-47f2-8503-b66092a86230-kube-api-access-r8lcx") pod "network-check-target-zsq7v" (UID: "3e0c7b26-181b-47f2-8503-b66092a86230") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:34:26.036301 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:26.036260 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bq6hn" Jul 07 14:34:26.036891 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:26.036408 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bq6hn" podUID="686ecb3c-7c9a-4ac1-ac27-4430651b0870" Jul 07 14:34:26.036891 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:26.036495 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpmtl" Jul 07 14:34:26.036891 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:26.036549 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zsq7v" Jul 07 14:34:26.036891 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:26.036658 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpmtl" podUID="e3bbd20c-73f5-4d24-aed9-145e4b89a590" Jul 07 14:34:26.036891 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:26.036790 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zsq7v" podUID="3e0c7b26-181b-47f2-8503-b66092a86230" Jul 07 14:34:28.036792 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:28.036704 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpmtl" Jul 07 14:34:28.036792 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:28.036727 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bq6hn" Jul 07 14:34:28.037359 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:28.036887 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zsq7v" Jul 07 14:34:28.037359 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:28.036897 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpmtl" podUID="e3bbd20c-73f5-4d24-aed9-145e4b89a590" Jul 07 14:34:28.037359 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:28.036999 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zsq7v" podUID="3e0c7b26-181b-47f2-8503-b66092a86230" Jul 07 14:34:28.037359 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:28.037084 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bq6hn" podUID="686ecb3c-7c9a-4ac1-ac27-4430651b0870" Jul 07 14:34:28.656518 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:28.655919 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/686ecb3c-7c9a-4ac1-ac27-4430651b0870-original-pull-secret\") pod \"global-pull-secret-syncer-bq6hn\" (UID: \"686ecb3c-7c9a-4ac1-ac27-4430651b0870\") " pod="kube-system/global-pull-secret-syncer-bq6hn" Jul 07 14:34:28.656518 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:28.655979 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3bbd20c-73f5-4d24-aed9-145e4b89a590-metrics-certs\") pod \"network-metrics-daemon-lpmtl\" (UID: \"e3bbd20c-73f5-4d24-aed9-145e4b89a590\") " pod="openshift-multus/network-metrics-daemon-lpmtl" Jul 07 14:34:28.656518 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:28.656064 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 14:34:28.656518 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:28.656156 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/686ecb3c-7c9a-4ac1-ac27-4430651b0870-original-pull-secret podName:686ecb3c-7c9a-4ac1-ac27-4430651b0870 nodeName:}" failed. No retries permitted until 2026-07-07 14:34:36.656132964 +0000 UTC m=+18.080319423 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/686ecb3c-7c9a-4ac1-ac27-4430651b0870-original-pull-secret") pod "global-pull-secret-syncer-bq6hn" (UID: "686ecb3c-7c9a-4ac1-ac27-4430651b0870") : object "kube-system"/"original-pull-secret" not registered Jul 07 14:34:28.656518 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:28.656158 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:34:28.656518 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:28.656196 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3bbd20c-73f5-4d24-aed9-145e4b89a590-metrics-certs podName:e3bbd20c-73f5-4d24-aed9-145e4b89a590 nodeName:}" failed. No retries permitted until 2026-07-07 14:34:36.656188057 +0000 UTC m=+18.080374512 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3bbd20c-73f5-4d24-aed9-145e4b89a590-metrics-certs") pod "network-metrics-daemon-lpmtl" (UID: "e3bbd20c-73f5-4d24-aed9-145e4b89a590") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:34:28.757362 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:28.757305 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8lcx\" (UniqueName: \"kubernetes.io/projected/3e0c7b26-181b-47f2-8503-b66092a86230-kube-api-access-r8lcx\") pod \"network-check-target-zsq7v\" (UID: \"3e0c7b26-181b-47f2-8503-b66092a86230\") " pod="openshift-network-diagnostics/network-check-target-zsq7v" Jul 07 14:34:28.757548 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:28.757531 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 14:34:28.757605 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:28.757557 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 14:34:28.757605 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:28.757571 2555 projected.go:196] Error preparing data for projected volume kube-api-access-r8lcx for pod openshift-network-diagnostics/network-check-target-zsq7v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:34:28.757675 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:28.757637 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3e0c7b26-181b-47f2-8503-b66092a86230-kube-api-access-r8lcx podName:3e0c7b26-181b-47f2-8503-b66092a86230 nodeName:}" failed. No retries permitted until 2026-07-07 14:34:36.757616874 +0000 UTC m=+18.181803348 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-r8lcx" (UniqueName: "kubernetes.io/projected/3e0c7b26-181b-47f2-8503-b66092a86230-kube-api-access-r8lcx") pod "network-check-target-zsq7v" (UID: "3e0c7b26-181b-47f2-8503-b66092a86230") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:34:30.036504 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:30.036460 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpmtl" Jul 07 14:34:30.037051 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:30.036618 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpmtl" podUID="e3bbd20c-73f5-4d24-aed9-145e4b89a590" Jul 07 14:34:30.037051 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:30.036819 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bq6hn" Jul 07 14:34:30.037051 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:30.036903 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bq6hn" podUID="686ecb3c-7c9a-4ac1-ac27-4430651b0870" Jul 07 14:34:30.037051 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:30.036954 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zsq7v" Jul 07 14:34:30.037051 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:30.037021 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zsq7v" podUID="3e0c7b26-181b-47f2-8503-b66092a86230" Jul 07 14:34:32.036636 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:32.036594 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zsq7v" Jul 07 14:34:32.037109 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:32.036724 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zsq7v" podUID="3e0c7b26-181b-47f2-8503-b66092a86230" Jul 07 14:34:32.037178 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:32.037128 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpmtl" Jul 07 14:34:32.037269 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:32.037231 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpmtl" podUID="e3bbd20c-73f5-4d24-aed9-145e4b89a590" Jul 07 14:34:32.037326 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:32.037276 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bq6hn" Jul 07 14:34:32.037426 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:32.037389 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bq6hn" podUID="686ecb3c-7c9a-4ac1-ac27-4430651b0870" Jul 07 14:34:34.036635 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:34.036598 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpmtl" Jul 07 14:34:34.036635 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:34.036633 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zsq7v" Jul 07 14:34:34.037181 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:34.036633 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bq6hn" Jul 07 14:34:34.037181 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:34.036727 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpmtl" podUID="e3bbd20c-73f5-4d24-aed9-145e4b89a590" Jul 07 14:34:34.037181 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:34.036805 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zsq7v" podUID="3e0c7b26-181b-47f2-8503-b66092a86230" Jul 07 14:34:34.037181 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:34.036860 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bq6hn" podUID="686ecb3c-7c9a-4ac1-ac27-4430651b0870" Jul 07 14:34:36.036187 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:36.036148 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bq6hn" Jul 07 14:34:36.036728 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:36.036148 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpmtl" Jul 07 14:34:36.036728 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:36.036271 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bq6hn" podUID="686ecb3c-7c9a-4ac1-ac27-4430651b0870" Jul 07 14:34:36.036728 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:36.036367 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpmtl" podUID="e3bbd20c-73f5-4d24-aed9-145e4b89a590" Jul 07 14:34:36.036728 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:36.036150 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zsq7v" Jul 07 14:34:36.036728 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:36.036491 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zsq7v" podUID="3e0c7b26-181b-47f2-8503-b66092a86230" Jul 07 14:34:36.724746 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:36.724706 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3bbd20c-73f5-4d24-aed9-145e4b89a590-metrics-certs\") pod \"network-metrics-daemon-lpmtl\" (UID: \"e3bbd20c-73f5-4d24-aed9-145e4b89a590\") " pod="openshift-multus/network-metrics-daemon-lpmtl" Jul 07 14:34:36.724928 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:36.724822 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/686ecb3c-7c9a-4ac1-ac27-4430651b0870-original-pull-secret\") pod \"global-pull-secret-syncer-bq6hn\" (UID: \"686ecb3c-7c9a-4ac1-ac27-4430651b0870\") " pod="kube-system/global-pull-secret-syncer-bq6hn" Jul 07 14:34:36.724928 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:36.724866 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:34:36.724928 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:36.724893 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 14:34:36.725094 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:36.724936 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/686ecb3c-7c9a-4ac1-ac27-4430651b0870-original-pull-secret podName:686ecb3c-7c9a-4ac1-ac27-4430651b0870 nodeName:}" failed. No retries permitted until 2026-07-07 14:34:52.724919629 +0000 UTC m=+34.149106082 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/686ecb3c-7c9a-4ac1-ac27-4430651b0870-original-pull-secret") pod "global-pull-secret-syncer-bq6hn" (UID: "686ecb3c-7c9a-4ac1-ac27-4430651b0870") : object "kube-system"/"original-pull-secret" not registered Jul 07 14:34:36.725094 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:36.724953 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3bbd20c-73f5-4d24-aed9-145e4b89a590-metrics-certs podName:e3bbd20c-73f5-4d24-aed9-145e4b89a590 nodeName:}" failed. No retries permitted until 2026-07-07 14:34:52.724944892 +0000 UTC m=+34.149131355 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3bbd20c-73f5-4d24-aed9-145e4b89a590-metrics-certs") pod "network-metrics-daemon-lpmtl" (UID: "e3bbd20c-73f5-4d24-aed9-145e4b89a590") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:34:36.825718 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:36.825687 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8lcx\" (UniqueName: \"kubernetes.io/projected/3e0c7b26-181b-47f2-8503-b66092a86230-kube-api-access-r8lcx\") pod \"network-check-target-zsq7v\" (UID: \"3e0c7b26-181b-47f2-8503-b66092a86230\") " pod="openshift-network-diagnostics/network-check-target-zsq7v" Jul 07 14:34:36.825945 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:36.825859 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 14:34:36.825945 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:36.825877 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 14:34:36.825945 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:36.825887 2555 projected.go:196] Error preparing data for projected volume kube-api-access-r8lcx for pod openshift-network-diagnostics/network-check-target-zsq7v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:34:36.825945 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:36.825934 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3e0c7b26-181b-47f2-8503-b66092a86230-kube-api-access-r8lcx podName:3e0c7b26-181b-47f2-8503-b66092a86230 nodeName:}" failed. No retries permitted until 2026-07-07 14:34:52.825922262 +0000 UTC m=+34.250108715 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-r8lcx" (UniqueName: "kubernetes.io/projected/3e0c7b26-181b-47f2-8503-b66092a86230-kube-api-access-r8lcx") pod "network-check-target-zsq7v" (UID: "3e0c7b26-181b-47f2-8503-b66092a86230") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:34:38.036277 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:38.036242 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bq6hn" Jul 07 14:34:38.036776 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:38.036247 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zsq7v" Jul 07 14:34:38.036776 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:38.036364 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bq6hn" podUID="686ecb3c-7c9a-4ac1-ac27-4430651b0870" Jul 07 14:34:38.036776 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:38.036374 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpmtl" Jul 07 14:34:38.036776 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:38.036450 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zsq7v" podUID="3e0c7b26-181b-47f2-8503-b66092a86230" Jul 07 14:34:38.036776 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:38.036535 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpmtl" podUID="e3bbd20c-73f5-4d24-aed9-145e4b89a590" Jul 07 14:34:40.036054 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:40.036014 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zsq7v" Jul 07 14:34:40.036538 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:40.036019 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpmtl" Jul 07 14:34:40.036538 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:40.036146 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zsq7v" podUID="3e0c7b26-181b-47f2-8503-b66092a86230" Jul 07 14:34:40.036538 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:40.036023 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bq6hn" Jul 07 14:34:40.036538 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:40.036271 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpmtl" podUID="e3bbd20c-73f5-4d24-aed9-145e4b89a590" Jul 07 14:34:40.036538 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:40.036369 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bq6hn" podUID="686ecb3c-7c9a-4ac1-ac27-4430651b0870" Jul 07 14:34:42.036437 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:42.036224 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpmtl" Jul 07 14:34:42.037201 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:42.036250 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bq6hn" Jul 07 14:34:42.037201 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:42.036488 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpmtl" podUID="e3bbd20c-73f5-4d24-aed9-145e4b89a590" Jul 07 14:34:42.037201 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:42.036272 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zsq7v" Jul 07 14:34:42.037201 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:42.036572 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bq6hn" podUID="686ecb3c-7c9a-4ac1-ac27-4430651b0870" Jul 07 14:34:42.037201 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:42.036659 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zsq7v" podUID="3e0c7b26-181b-47f2-8503-b66092a86230" Jul 07 14:34:42.188902 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:42.188869 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" event={"ID":"303d9d00-aaf9-431e-9980-60fd60904fad","Type":"ContainerStarted","Data":"0eb0d218fc8ad728dfb417bcb759625136e90ca2a1570afccc8332f09a1785a7"} Jul 07 14:34:42.189142 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:42.188948 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" event={"ID":"303d9d00-aaf9-431e-9980-60fd60904fad","Type":"ContainerStarted","Data":"b2a27436b126ff099a2dc8bc385ac9028beffcebb86c7e95eab1d1cfba12bc6a"} Jul 07 14:34:42.189142 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:42.188992 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" event={"ID":"303d9d00-aaf9-431e-9980-60fd60904fad","Type":"ContainerStarted","Data":"ef70818251291f95188b7a6e687a3002b8656bb0c5f4716d90622600e7fa34cc"} Jul 07 14:34:42.189142 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:42.189031 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" event={"ID":"303d9d00-aaf9-431e-9980-60fd60904fad","Type":"ContainerStarted","Data":"fd518fa69120244b3cf0a6987b63b885f6e2c914a7b942880afb4dec9abff6ad"} Jul 07 14:34:42.189142 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:42.189071 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" event={"ID":"303d9d00-aaf9-431e-9980-60fd60904fad","Type":"ContainerStarted","Data":"ddb7e575b936d29fcb7ad15d23c169e0bb65fb5467653c855e5e19462be8752e"} Jul 07 14:34:42.189142 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:42.189105 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" event={"ID":"303d9d00-aaf9-431e-9980-60fd60904fad","Type":"ContainerStarted","Data":"da127cbbdcc2108ad8f76106a6d225eeeebf5ddc41c88f7e427d8c96d3e8876f"} Jul 07 14:34:42.190283 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:42.190256 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bfcr2" event={"ID":"f5f5ff03-a3bc-4758-8023-e023ab4ed3b3","Type":"ContainerStarted","Data":"66a849543805809678cb1e53a705ddd9b919063b984a8d8f3a6053d2cedf4f29"} Jul 07 14:34:42.191602 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:42.191581 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4zclx" event={"ID":"3ea25b41-29d9-4e81-b4d5-6252d76a8e66","Type":"ContainerStarted","Data":"b23fdc4368538cf83c12018e113f00d5df46bb3f74fa0dea6b340b03a4457f72"} Jul 07 14:34:42.192969 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:42.192939 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vkwws" event={"ID":"7d94747d-5e8a-4eab-8234-969f7dfbaee5","Type":"ContainerStarted","Data":"0f3f5bef1f102d0c4694f306ee62cf7ce482e319006265191cc8959335a44da8"} Jul 07 14:34:42.194421 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:42.194401 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-czppd" event={"ID":"553f749a-649c-41c0-aa39-a0cb13725ca8","Type":"ContainerStarted","Data":"d3448eb79e9b3fab9499040e8bb8eee082565bd580da0f15dfac20b66a467cc1"} Jul 07 14:34:42.195856 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:42.195838 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" event={"ID":"aaf88254-ff84-430f-b609-debc1b4b170c","Type":"ContainerStarted","Data":"a93142f8182acf26f58d887b267f39b517d5d27d3c5b5b231efec86ad6c286b7"} Jul 07 14:34:42.197371 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:42.197350 2555 generic.go:356] "Generic (PLEG): container finished" podID="89159896-e9f7-42e0-93e7-23122a296d8c" containerID="573990c9c67fcf9dca511871681c3643d3f4ca5afe20debdc7bc4a3947d6b8e7" exitCode=0 Jul 07 14:34:42.197454 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:42.197428 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9t7xp" event={"ID":"89159896-e9f7-42e0-93e7-23122a296d8c","Type":"ContainerDied","Data":"573990c9c67fcf9dca511871681c3643d3f4ca5afe20debdc7bc4a3947d6b8e7"} Jul 07 14:34:42.198996 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:42.198980 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb4h5" event={"ID":"ff6cf858-eb9e-4adc-b4ae-9671841ef488","Type":"ContainerStarted","Data":"b23bbbf0a3f0860cc40f5a7da93bdd716c17b16555462e118248ea1e05564b70"} Jul 07 14:34:42.208803 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:42.208766 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-bfcr2" podStartSLOduration=3.145234265 podStartE2EDuration="22.20873746s" totalImagesPullingTime="19.063503195s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:34:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:34:42.208405404 +0000 UTC m=+23.632591879" watchObservedRunningTime="2026-07-07 14:34:42.20873746 +0000 UTC m=+23.632923935" Jul 07 14:34:42.224299 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:42.224266 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vs7xn" podStartSLOduration=4.243843444 podStartE2EDuration="23.22425355s" totalImagesPullingTime="18.980410106s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:34:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:34:42.223956214 +0000 UTC m=+23.648142705" watchObservedRunningTime="2026-07-07 14:34:42.22425355 +0000 UTC m=+23.648440347" Jul 07 14:34:42.257963 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:42.257917 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vkwws" podStartSLOduration=3.253378881 podStartE2EDuration="22.257902708s" totalImagesPullingTime="19.004523827s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:34:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:34:42.257525401 +0000 UTC m=+23.681711875" watchObservedRunningTime="2026-07-07 14:34:42.257902708 +0000 UTC m=+23.682089182" Jul 07 14:34:42.289983 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:42.289911 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4zclx" podStartSLOduration=12.325981727 podStartE2EDuration="23.289896456s" totalImagesPullingTime="10.963914729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:34:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:34:42.274640182 +0000 UTC m=+23.698826658" watchObservedRunningTime="2026-07-07 14:34:42.289896456 +0000 UTC m=+23.714082925" Jul 07 14:34:42.290262 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:42.290244 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-czppd" podStartSLOduration=3.313678621 podStartE2EDuration="22.290237541s" totalImagesPullingTime="18.97655892s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:34:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:34:42.289617631 +0000 UTC m=+23.713804106" watchObservedRunningTime="2026-07-07 14:34:42.290237541 +0000 UTC m=+23.714424015" Jul 07 14:34:42.585158 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:42.585132 2555 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 07 14:34:42.943518 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:42.943372 2555 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T14:34:42.585148707Z","UUID":"68b617b5-8747-474f-8159-724603d3bbec","Handler":null,"Name":"","Endpoint":""} Jul 07 14:34:42.945600 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:42.945577 2555 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 07 14:34:42.945744 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:42.945623 2555 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 07 14:34:43.204267 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:43.204193 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb4h5" event={"ID":"ff6cf858-eb9e-4adc-b4ae-9671841ef488","Type":"ContainerStarted","Data":"6e5696a24032b07090b39b30cc91bea51a0c731f83d223bc0d0802f3fae97901"} Jul 07 14:34:43.205966 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:43.205936 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mj6xk" event={"ID":"c06aa1d3-1eb1-4a64-9677-2b7328a9cff9","Type":"ContainerStarted","Data":"2dee953c8a728c7c191df0e6b61634871ea54290b1fcdd1af2e8ddfc6af7a413"} Jul 07 14:34:43.221382 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:43.221333 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-mj6xk" podStartSLOduration=4.208451306 podStartE2EDuration="23.221313366s" totalImagesPullingTime="19.01286206s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:34:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:34:43.221046398 +0000 UTC m=+24.645232874" watchObservedRunningTime="2026-07-07 14:34:43.221313366 +0000 UTC m=+24.645499841" Jul 07 14:34:43.470737 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:43.470654 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4zclx" Jul 07 14:34:44.036572 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:44.036537 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zsq7v" Jul 07 14:34:44.036805 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:44.036537 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bq6hn" Jul 07 14:34:44.036805 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:44.036689 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zsq7v" podUID="3e0c7b26-181b-47f2-8503-b66092a86230" Jul 07 14:34:44.036805 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:44.036735 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bq6hn" podUID="686ecb3c-7c9a-4ac1-ac27-4430651b0870" Jul 07 14:34:44.036805 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:44.036545 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpmtl" Jul 07 14:34:44.036973 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:44.036880 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpmtl" podUID="e3bbd20c-73f5-4d24-aed9-145e4b89a590" Jul 07 14:34:44.212197 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:44.212111 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" event={"ID":"303d9d00-aaf9-431e-9980-60fd60904fad","Type":"ContainerStarted","Data":"2c0c8778f449027ec81bc80e4ac1cc2f1859ad671a43a94a4b8c4737b8744bf4"} Jul 07 14:34:45.609610 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:45.609406 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4zclx" Jul 07 14:34:45.610264 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:45.610236 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4zclx" Jul 07 14:34:46.036278 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:46.036185 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bq6hn" Jul 07 14:34:46.036529 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:46.036185 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpmtl" Jul 07 14:34:46.036529 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:46.036318 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bq6hn" podUID="686ecb3c-7c9a-4ac1-ac27-4430651b0870" Jul 07 14:34:46.036529 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:46.036188 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zsq7v" Jul 07 14:34:46.036529 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:46.036407 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpmtl" podUID="e3bbd20c-73f5-4d24-aed9-145e4b89a590" Jul 07 14:34:46.036529 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:46.036480 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zsq7v" podUID="3e0c7b26-181b-47f2-8503-b66092a86230" Jul 07 14:34:46.217517 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:46.217482 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4zclx" Jul 07 14:34:47.220369 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:47.220137 2555 generic.go:356] "Generic (PLEG): container finished" podID="89159896-e9f7-42e0-93e7-23122a296d8c" containerID="3e04cf1633810538a97205bfe281b58db563c27a745a3d2d6777b2bb9dfe5e13" exitCode=0 Jul 07 14:34:47.220369 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:47.220227 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9t7xp" event={"ID":"89159896-e9f7-42e0-93e7-23122a296d8c","Type":"ContainerDied","Data":"3e04cf1633810538a97205bfe281b58db563c27a745a3d2d6777b2bb9dfe5e13"} Jul 07 14:34:47.222584 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:47.222555 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb4h5" event={"ID":"ff6cf858-eb9e-4adc-b4ae-9671841ef488","Type":"ContainerStarted","Data":"b113a625dc6621795e3dbb4c8178665c57882fb84ae3ec347d347ecaab642dc7"} Jul 07 14:34:47.226193 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:47.226161 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" event={"ID":"303d9d00-aaf9-431e-9980-60fd60904fad","Type":"ContainerStarted","Data":"df9864938553be4d82201fa9251b2f7ec53f2a23ec2e42be95d332be89b947bd"} Jul 07 14:34:47.263068 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:47.263003 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sb4h5" podStartSLOduration=3.364256217 podStartE2EDuration="27.262985533s" totalImagesPullingTime="23.898729316s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:34:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:34:47.262935828 +0000 UTC m=+28.687122302" watchObservedRunningTime="2026-07-07 14:34:47.262985533 +0000 UTC m=+28.687172008" Jul 07 14:34:47.289050 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:47.289002 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" podStartSLOduration=8.216841908 podStartE2EDuration="27.288985873s" totalImagesPullingTime="19.072143965s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:34:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:34:47.287638455 +0000 UTC m=+28.711824929" watchObservedRunningTime="2026-07-07 14:34:47.288985873 +0000 UTC m=+28.713172347" Jul 07 14:34:48.036608 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:48.036576 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpmtl" Jul 07 14:34:48.036888 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:48.036619 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bq6hn" Jul 07 14:34:48.036888 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:48.036689 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zsq7v" Jul 07 14:34:48.036888 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:48.036709 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpmtl" podUID="e3bbd20c-73f5-4d24-aed9-145e4b89a590" Jul 07 14:34:48.036888 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:48.036832 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zsq7v" podUID="3e0c7b26-181b-47f2-8503-b66092a86230" Jul 07 14:34:48.037100 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:48.036891 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bq6hn" podUID="686ecb3c-7c9a-4ac1-ac27-4430651b0870" Jul 07 14:34:48.229608 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:48.229570 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:48.230091 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:48.229648 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:48.230091 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:48.229693 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:48.247342 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:48.247308 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:48.247764 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:48.247725 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:34:48.471937 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:48.471840 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bq6hn"] Jul 07 14:34:48.471997 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:48.471945 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bq6hn" Jul 07 14:34:48.472037 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:48.472024 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bq6hn" podUID="686ecb3c-7c9a-4ac1-ac27-4430651b0870" Jul 07 14:34:48.475452 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:48.475429 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lpmtl"] Jul 07 14:34:48.475576 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:48.475520 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpmtl" Jul 07 14:34:48.475641 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:48.475604 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpmtl" podUID="e3bbd20c-73f5-4d24-aed9-145e4b89a590" Jul 07 14:34:48.475967 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:48.475945 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zsq7v"] Jul 07 14:34:48.476043 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:48.476015 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zsq7v" Jul 07 14:34:48.476101 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:48.476086 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zsq7v" podUID="3e0c7b26-181b-47f2-8503-b66092a86230" Jul 07 14:34:49.233422 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:49.233227 2555 generic.go:356] "Generic (PLEG): container finished" podID="89159896-e9f7-42e0-93e7-23122a296d8c" containerID="ee51379db0c39163e5eb97ee945d19c80da9e7b63910abc4504d6c1ceb82f299" exitCode=0 Jul 07 14:34:49.233422 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:49.233314 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9t7xp" event={"ID":"89159896-e9f7-42e0-93e7-23122a296d8c","Type":"ContainerDied","Data":"ee51379db0c39163e5eb97ee945d19c80da9e7b63910abc4504d6c1ceb82f299"} Jul 07 14:34:50.036158 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:50.036123 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bq6hn" Jul 07 14:34:50.036158 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:50.036163 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zsq7v" Jul 07 14:34:50.036421 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:50.036139 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpmtl" Jul 07 14:34:50.036421 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:50.036256 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bq6hn" podUID="686ecb3c-7c9a-4ac1-ac27-4430651b0870" Jul 07 14:34:50.036421 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:50.036315 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpmtl" podUID="e3bbd20c-73f5-4d24-aed9-145e4b89a590" Jul 07 14:34:50.036421 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:50.036365 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zsq7v" podUID="3e0c7b26-181b-47f2-8503-b66092a86230" Jul 07 14:34:51.241126 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:51.241046 2555 generic.go:356] "Generic (PLEG): container finished" podID="89159896-e9f7-42e0-93e7-23122a296d8c" containerID="0e2e1bdc10b9ef8ac5c90b743ce9277eaf7553cca81d492f9aab4e20a7101a88" exitCode=0 Jul 07 14:34:51.241126 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:51.241102 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9t7xp" event={"ID":"89159896-e9f7-42e0-93e7-23122a296d8c","Type":"ContainerDied","Data":"0e2e1bdc10b9ef8ac5c90b743ce9277eaf7553cca81d492f9aab4e20a7101a88"} Jul 07 14:34:52.035969 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:52.035927 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zsq7v" Jul 07 14:34:52.036132 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:52.035927 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bq6hn" Jul 07 14:34:52.036132 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:52.036044 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zsq7v" podUID="3e0c7b26-181b-47f2-8503-b66092a86230" Jul 07 14:34:52.036216 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:52.036154 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bq6hn" podUID="686ecb3c-7c9a-4ac1-ac27-4430651b0870" Jul 07 14:34:52.036216 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:52.035930 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpmtl" Jul 07 14:34:52.036295 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:52.036277 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpmtl" podUID="e3bbd20c-73f5-4d24-aed9-145e4b89a590" Jul 07 14:34:52.740357 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:52.740314 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/686ecb3c-7c9a-4ac1-ac27-4430651b0870-original-pull-secret\") pod \"global-pull-secret-syncer-bq6hn\" (UID: \"686ecb3c-7c9a-4ac1-ac27-4430651b0870\") " pod="kube-system/global-pull-secret-syncer-bq6hn" Jul 07 14:34:52.740357 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:52.740360 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3bbd20c-73f5-4d24-aed9-145e4b89a590-metrics-certs\") pod \"network-metrics-daemon-lpmtl\" (UID: \"e3bbd20c-73f5-4d24-aed9-145e4b89a590\") " pod="openshift-multus/network-metrics-daemon-lpmtl" Jul 07 14:34:52.740973 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:52.740489 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 14:34:52.740973 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:52.740557 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/686ecb3c-7c9a-4ac1-ac27-4430651b0870-original-pull-secret podName:686ecb3c-7c9a-4ac1-ac27-4430651b0870 nodeName:}" failed. No retries permitted until 2026-07-07 14:35:24.740542028 +0000 UTC m=+66.164728481 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/686ecb3c-7c9a-4ac1-ac27-4430651b0870-original-pull-secret") pod "global-pull-secret-syncer-bq6hn" (UID: "686ecb3c-7c9a-4ac1-ac27-4430651b0870") : object "kube-system"/"original-pull-secret" not registered Jul 07 14:34:52.740973 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:52.740496 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:34:52.740973 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:52.740641 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3bbd20c-73f5-4d24-aed9-145e4b89a590-metrics-certs podName:e3bbd20c-73f5-4d24-aed9-145e4b89a590 nodeName:}" failed. No retries permitted until 2026-07-07 14:35:24.740628468 +0000 UTC m=+66.164814922 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3bbd20c-73f5-4d24-aed9-145e4b89a590-metrics-certs") pod "network-metrics-daemon-lpmtl" (UID: "e3bbd20c-73f5-4d24-aed9-145e4b89a590") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:34:52.840684 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:52.840646 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8lcx\" (UniqueName: \"kubernetes.io/projected/3e0c7b26-181b-47f2-8503-b66092a86230-kube-api-access-r8lcx\") pod \"network-check-target-zsq7v\" (UID: \"3e0c7b26-181b-47f2-8503-b66092a86230\") " pod="openshift-network-diagnostics/network-check-target-zsq7v" Jul 07 14:34:52.840901 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:52.840803 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 14:34:52.840901 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:52.840828 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 14:34:52.840901 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:52.840841 2555 projected.go:196] Error preparing data for projected volume kube-api-access-r8lcx for pod openshift-network-diagnostics/network-check-target-zsq7v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:34:52.841072 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:52.840904 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3e0c7b26-181b-47f2-8503-b66092a86230-kube-api-access-r8lcx podName:3e0c7b26-181b-47f2-8503-b66092a86230 nodeName:}" failed. No retries permitted until 2026-07-07 14:35:24.840884109 +0000 UTC m=+66.265070575 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-r8lcx" (UniqueName: "kubernetes.io/projected/3e0c7b26-181b-47f2-8503-b66092a86230-kube-api-access-r8lcx") pod "network-check-target-zsq7v" (UID: "3e0c7b26-181b-47f2-8503-b66092a86230") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:34:53.925941 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:53.925909 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-19.ec2.internal" event="NodeReady" Jul 07 14:34:53.926478 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:53.926044 2555 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 07 14:34:53.974913 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:53.974843 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xv4vf"] Jul 07 14:34:53.979783 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:53.979745 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9pd7h"] Jul 07 14:34:53.979913 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:53.979842 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xv4vf" Jul 07 14:34:53.983217 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:53.983166 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ntb78\"" Jul 07 14:34:53.983310 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:53.983250 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 07 14:34:53.983392 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:53.983374 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9pd7h" Jul 07 14:34:53.983526 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:53.983505 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 07 14:34:53.983526 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:53.983523 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 07 14:34:53.985918 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:53.985899 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-v5tsw\"" Jul 07 14:34:53.985918 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:53.985909 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 07 14:34:53.986344 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:53.986308 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 07 14:34:53.987384 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:53.987364 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xv4vf"] Jul 07 14:34:53.992666 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:53.992647 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9pd7h"] Jul 07 14:34:54.035959 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:54.035933 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpmtl" Jul 07 14:34:54.036140 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:54.036071 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zsq7v" Jul 07 14:34:54.036299 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:54.036285 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bq6hn" Jul 07 14:34:54.038421 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:54.038400 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 14:34:54.038725 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:54.038708 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-v8hh9\"" Jul 07 14:34:54.038828 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:54.038775 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 14:34:54.038884 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:54.038708 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dhggt\"" Jul 07 14:34:54.038935 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:54.038906 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 14:34:54.038988 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:54.038933 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 14:34:54.150318 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:54.150285 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8c25e009-ed68-418d-b5c1-02d8fa948f74-cert\") pod \"ingress-canary-xv4vf\" (UID: \"8c25e009-ed68-418d-b5c1-02d8fa948f74\") " pod="openshift-ingress-canary/ingress-canary-xv4vf" Jul 07 14:34:54.150318 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:54.150319 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c447425e-fb20-4ac8-b7e7-ed07d60953e5-metrics-tls\") pod \"dns-default-9pd7h\" (UID: \"c447425e-fb20-4ac8-b7e7-ed07d60953e5\") " pod="openshift-dns/dns-default-9pd7h" Jul 07 14:34:54.150561 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:54.150431 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c447425e-fb20-4ac8-b7e7-ed07d60953e5-config-volume\") pod \"dns-default-9pd7h\" (UID: \"c447425e-fb20-4ac8-b7e7-ed07d60953e5\") " pod="openshift-dns/dns-default-9pd7h" Jul 07 14:34:54.150561 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:54.150492 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c447425e-fb20-4ac8-b7e7-ed07d60953e5-tmp-dir\") pod \"dns-default-9pd7h\" (UID: \"c447425e-fb20-4ac8-b7e7-ed07d60953e5\") " pod="openshift-dns/dns-default-9pd7h" Jul 07 14:34:54.150561 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:54.150537 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6nhv\" (UniqueName: \"kubernetes.io/projected/c447425e-fb20-4ac8-b7e7-ed07d60953e5-kube-api-access-g6nhv\") pod \"dns-default-9pd7h\" (UID: \"c447425e-fb20-4ac8-b7e7-ed07d60953e5\") " pod="openshift-dns/dns-default-9pd7h" Jul 07 14:34:54.150705 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:54.150578 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtvvn\" (UniqueName: \"kubernetes.io/projected/8c25e009-ed68-418d-b5c1-02d8fa948f74-kube-api-access-rtvvn\") pod \"ingress-canary-xv4vf\" (UID: \"8c25e009-ed68-418d-b5c1-02d8fa948f74\") " pod="openshift-ingress-canary/ingress-canary-xv4vf" Jul 07 14:34:54.251561 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:54.251467 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8c25e009-ed68-418d-b5c1-02d8fa948f74-cert\") pod \"ingress-canary-xv4vf\" (UID: \"8c25e009-ed68-418d-b5c1-02d8fa948f74\") " pod="openshift-ingress-canary/ingress-canary-xv4vf" Jul 07 14:34:54.251561 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:54.251508 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c447425e-fb20-4ac8-b7e7-ed07d60953e5-metrics-tls\") pod \"dns-default-9pd7h\" (UID: \"c447425e-fb20-4ac8-b7e7-ed07d60953e5\") " pod="openshift-dns/dns-default-9pd7h" Jul 07 14:34:54.251799 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:54.251576 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c447425e-fb20-4ac8-b7e7-ed07d60953e5-config-volume\") pod \"dns-default-9pd7h\" (UID: \"c447425e-fb20-4ac8-b7e7-ed07d60953e5\") " pod="openshift-dns/dns-default-9pd7h" Jul 07 14:34:54.251799 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:54.251610 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c447425e-fb20-4ac8-b7e7-ed07d60953e5-tmp-dir\") pod \"dns-default-9pd7h\" (UID: \"c447425e-fb20-4ac8-b7e7-ed07d60953e5\") " pod="openshift-dns/dns-default-9pd7h" Jul 07 14:34:54.251799 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:54.251634 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6nhv\" (UniqueName: \"kubernetes.io/projected/c447425e-fb20-4ac8-b7e7-ed07d60953e5-kube-api-access-g6nhv\") pod \"dns-default-9pd7h\" (UID: \"c447425e-fb20-4ac8-b7e7-ed07d60953e5\") " pod="openshift-dns/dns-default-9pd7h" Jul 07 14:34:54.251799 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:54.251665 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtvvn\" (UniqueName: \"kubernetes.io/projected/8c25e009-ed68-418d-b5c1-02d8fa948f74-kube-api-access-rtvvn\") pod \"ingress-canary-xv4vf\" (UID: \"8c25e009-ed68-418d-b5c1-02d8fa948f74\") " pod="openshift-ingress-canary/ingress-canary-xv4vf" Jul 07 14:34:54.251799 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:54.251692 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 14:34:54.251799 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:54.251788 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c447425e-fb20-4ac8-b7e7-ed07d60953e5-metrics-tls podName:c447425e-fb20-4ac8-b7e7-ed07d60953e5 nodeName:}" failed. No retries permitted until 2026-07-07 14:34:54.751744084 +0000 UTC m=+36.175930542 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c447425e-fb20-4ac8-b7e7-ed07d60953e5-metrics-tls") pod "dns-default-9pd7h" (UID: "c447425e-fb20-4ac8-b7e7-ed07d60953e5") : secret "dns-default-metrics-tls" not found Jul 07 14:34:54.252203 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:54.252177 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c447425e-fb20-4ac8-b7e7-ed07d60953e5-tmp-dir\") pod \"dns-default-9pd7h\" (UID: \"c447425e-fb20-4ac8-b7e7-ed07d60953e5\") " pod="openshift-dns/dns-default-9pd7h" Jul 07 14:34:54.252300 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:54.252261 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c447425e-fb20-4ac8-b7e7-ed07d60953e5-config-volume\") pod \"dns-default-9pd7h\" (UID: \"c447425e-fb20-4ac8-b7e7-ed07d60953e5\") " pod="openshift-dns/dns-default-9pd7h" Jul 07 14:34:54.256221 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:54.256201 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8c25e009-ed68-418d-b5c1-02d8fa948f74-cert\") pod \"ingress-canary-xv4vf\" (UID: \"8c25e009-ed68-418d-b5c1-02d8fa948f74\") " pod="openshift-ingress-canary/ingress-canary-xv4vf" Jul 07 14:34:54.262637 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:54.262614 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtvvn\" (UniqueName: \"kubernetes.io/projected/8c25e009-ed68-418d-b5c1-02d8fa948f74-kube-api-access-rtvvn\") pod \"ingress-canary-xv4vf\" (UID: \"8c25e009-ed68-418d-b5c1-02d8fa948f74\") " pod="openshift-ingress-canary/ingress-canary-xv4vf" Jul 07 14:34:54.262815 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:54.262794 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6nhv\" (UniqueName: \"kubernetes.io/projected/c447425e-fb20-4ac8-b7e7-ed07d60953e5-kube-api-access-g6nhv\") pod \"dns-default-9pd7h\" (UID: \"c447425e-fb20-4ac8-b7e7-ed07d60953e5\") " pod="openshift-dns/dns-default-9pd7h" Jul 07 14:34:54.293618 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:54.293589 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xv4vf" Jul 07 14:34:54.754863 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:54.754824 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c447425e-fb20-4ac8-b7e7-ed07d60953e5-metrics-tls\") pod \"dns-default-9pd7h\" (UID: \"c447425e-fb20-4ac8-b7e7-ed07d60953e5\") " pod="openshift-dns/dns-default-9pd7h" Jul 07 14:34:54.755056 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:54.754993 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 14:34:54.755112 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:54.755075 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c447425e-fb20-4ac8-b7e7-ed07d60953e5-metrics-tls podName:c447425e-fb20-4ac8-b7e7-ed07d60953e5 nodeName:}" failed. No retries permitted until 2026-07-07 14:34:55.755053134 +0000 UTC m=+37.179239593 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c447425e-fb20-4ac8-b7e7-ed07d60953e5-metrics-tls") pod "dns-default-9pd7h" (UID: "c447425e-fb20-4ac8-b7e7-ed07d60953e5") : secret "dns-default-metrics-tls" not found Jul 07 14:34:55.392509 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:55.392364 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xv4vf"] Jul 07 14:34:55.395540 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:34:55.395515 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c25e009_ed68_418d_b5c1_02d8fa948f74.slice/crio-3da6500efef10afc75b1f894afdef9c2dab09d073899f8cca6c47a202d05ba2e WatchSource:0}: Error finding container 3da6500efef10afc75b1f894afdef9c2dab09d073899f8cca6c47a202d05ba2e: Status 404 returned error can't find the container with id 3da6500efef10afc75b1f894afdef9c2dab09d073899f8cca6c47a202d05ba2e Jul 07 14:34:55.762398 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:55.762364 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c447425e-fb20-4ac8-b7e7-ed07d60953e5-metrics-tls\") pod \"dns-default-9pd7h\" (UID: \"c447425e-fb20-4ac8-b7e7-ed07d60953e5\") " pod="openshift-dns/dns-default-9pd7h" Jul 07 14:34:55.762560 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:55.762509 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 14:34:55.762604 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:55.762567 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c447425e-fb20-4ac8-b7e7-ed07d60953e5-metrics-tls podName:c447425e-fb20-4ac8-b7e7-ed07d60953e5 nodeName:}" failed. No retries permitted until 2026-07-07 14:34:57.762551634 +0000 UTC m=+39.186738092 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c447425e-fb20-4ac8-b7e7-ed07d60953e5-metrics-tls") pod "dns-default-9pd7h" (UID: "c447425e-fb20-4ac8-b7e7-ed07d60953e5") : secret "dns-default-metrics-tls" not found Jul 07 14:34:56.256647 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:56.256618 2555 generic.go:356] "Generic (PLEG): container finished" podID="89159896-e9f7-42e0-93e7-23122a296d8c" containerID="d1f57719ac619bb489b5d54fce628523ca1b085ef8d411247c3c7104f4ae788f" exitCode=0 Jul 07 14:34:56.256842 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:56.256700 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9t7xp" event={"ID":"89159896-e9f7-42e0-93e7-23122a296d8c","Type":"ContainerDied","Data":"d1f57719ac619bb489b5d54fce628523ca1b085ef8d411247c3c7104f4ae788f"} Jul 07 14:34:56.257936 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:56.257909 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xv4vf" event={"ID":"8c25e009-ed68-418d-b5c1-02d8fa948f74","Type":"ContainerStarted","Data":"3da6500efef10afc75b1f894afdef9c2dab09d073899f8cca6c47a202d05ba2e"} Jul 07 14:34:57.264206 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:57.264169 2555 generic.go:356] "Generic (PLEG): container finished" podID="89159896-e9f7-42e0-93e7-23122a296d8c" containerID="c21a8f4fe012a7be2b53ae3d2e538cd982602754e4a369d76596860d7ff00f57" exitCode=0 Jul 07 14:34:57.264667 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:57.264244 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9t7xp" event={"ID":"89159896-e9f7-42e0-93e7-23122a296d8c","Type":"ContainerDied","Data":"c21a8f4fe012a7be2b53ae3d2e538cd982602754e4a369d76596860d7ff00f57"} Jul 07 14:34:57.777644 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:57.777470 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c447425e-fb20-4ac8-b7e7-ed07d60953e5-metrics-tls\") pod \"dns-default-9pd7h\" (UID: \"c447425e-fb20-4ac8-b7e7-ed07d60953e5\") " pod="openshift-dns/dns-default-9pd7h" Jul 07 14:34:57.777835 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:57.777610 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 14:34:57.777835 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:34:57.777712 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c447425e-fb20-4ac8-b7e7-ed07d60953e5-metrics-tls podName:c447425e-fb20-4ac8-b7e7-ed07d60953e5 nodeName:}" failed. No retries permitted until 2026-07-07 14:35:01.777696899 +0000 UTC m=+43.201883352 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c447425e-fb20-4ac8-b7e7-ed07d60953e5-metrics-tls") pod "dns-default-9pd7h" (UID: "c447425e-fb20-4ac8-b7e7-ed07d60953e5") : secret "dns-default-metrics-tls" not found Jul 07 14:34:58.269238 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:58.269158 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9t7xp" event={"ID":"89159896-e9f7-42e0-93e7-23122a296d8c","Type":"ContainerStarted","Data":"8d045f1dfb74c3fedb0a4b64a8d81a0544fcf079f2bb5592595f43c962a1b8b5"} Jul 07 14:34:58.270578 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:58.270550 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xv4vf" event={"ID":"8c25e009-ed68-418d-b5c1-02d8fa948f74","Type":"ContainerStarted","Data":"90d986232a2662cda5a2ae505639b8185b548643957c562a1758530c5ec71d76"} Jul 07 14:34:58.295768 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:58.295708 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9t7xp" podStartSLOduration=8.170272101 podStartE2EDuration="38.295691995s" totalImagesPullingTime="30.125419894s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:34:20 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:34:58.293895311 +0000 UTC m=+39.718081801" watchObservedRunningTime="2026-07-07 14:34:58.295691995 +0000 UTC m=+39.719878469" Jul 07 14:34:58.313273 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:34:58.313236 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xv4vf" podStartSLOduration=2.826425381 podStartE2EDuration="5.313220511s" totalImagesPullingTime="2.48679513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:34:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:34:58.31276945 +0000 UTC m=+39.736955918" watchObservedRunningTime="2026-07-07 14:34:58.313220511 +0000 UTC m=+39.737406989" Jul 07 14:35:01.808151 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:35:01.808116 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c447425e-fb20-4ac8-b7e7-ed07d60953e5-metrics-tls\") pod \"dns-default-9pd7h\" (UID: \"c447425e-fb20-4ac8-b7e7-ed07d60953e5\") " pod="openshift-dns/dns-default-9pd7h" Jul 07 14:35:01.808600 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:35:01.808270 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 14:35:01.808600 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:35:01.808334 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c447425e-fb20-4ac8-b7e7-ed07d60953e5-metrics-tls podName:c447425e-fb20-4ac8-b7e7-ed07d60953e5 nodeName:}" failed. No retries permitted until 2026-07-07 14:35:09.808319996 +0000 UTC m=+51.232506452 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c447425e-fb20-4ac8-b7e7-ed07d60953e5-metrics-tls") pod "dns-default-9pd7h" (UID: "c447425e-fb20-4ac8-b7e7-ed07d60953e5") : secret "dns-default-metrics-tls" not found Jul 07 14:35:09.860499 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:35:09.860460 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c447425e-fb20-4ac8-b7e7-ed07d60953e5-metrics-tls\") pod \"dns-default-9pd7h\" (UID: \"c447425e-fb20-4ac8-b7e7-ed07d60953e5\") " pod="openshift-dns/dns-default-9pd7h" Jul 07 14:35:09.860985 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:35:09.860627 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 14:35:09.860985 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:35:09.860702 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c447425e-fb20-4ac8-b7e7-ed07d60953e5-metrics-tls podName:c447425e-fb20-4ac8-b7e7-ed07d60953e5 nodeName:}" failed. No retries permitted until 2026-07-07 14:35:25.860686271 +0000 UTC m=+67.284872744 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c447425e-fb20-4ac8-b7e7-ed07d60953e5-metrics-tls") pod "dns-default-9pd7h" (UID: "c447425e-fb20-4ac8-b7e7-ed07d60953e5") : secret "dns-default-metrics-tls" not found Jul 07 14:35:20.252564 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:35:20.252535 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-59hn4" Jul 07 14:35:24.764489 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:35:24.764445 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/686ecb3c-7c9a-4ac1-ac27-4430651b0870-original-pull-secret\") pod \"global-pull-secret-syncer-bq6hn\" (UID: \"686ecb3c-7c9a-4ac1-ac27-4430651b0870\") " pod="kube-system/global-pull-secret-syncer-bq6hn" Jul 07 14:35:24.764489 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:35:24.764494 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3bbd20c-73f5-4d24-aed9-145e4b89a590-metrics-certs\") pod \"network-metrics-daemon-lpmtl\" (UID: \"e3bbd20c-73f5-4d24-aed9-145e4b89a590\") " pod="openshift-multus/network-metrics-daemon-lpmtl" Jul 07 14:35:24.767229 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:35:24.767208 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 14:35:24.767300 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:35:24.767250 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 14:35:24.775262 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:35:24.775242 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 07 14:35:24.775325 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:35:24.775310 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3bbd20c-73f5-4d24-aed9-145e4b89a590-metrics-certs podName:e3bbd20c-73f5-4d24-aed9-145e4b89a590 nodeName:}" failed. No retries permitted until 2026-07-07 14:36:28.775293979 +0000 UTC m=+130.199480436 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3bbd20c-73f5-4d24-aed9-145e4b89a590-metrics-certs") pod "network-metrics-daemon-lpmtl" (UID: "e3bbd20c-73f5-4d24-aed9-145e4b89a590") : secret "metrics-daemon-secret" not found Jul 07 14:35:24.778927 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:35:24.778911 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/686ecb3c-7c9a-4ac1-ac27-4430651b0870-original-pull-secret\") pod \"global-pull-secret-syncer-bq6hn\" (UID: \"686ecb3c-7c9a-4ac1-ac27-4430651b0870\") " pod="kube-system/global-pull-secret-syncer-bq6hn" Jul 07 14:35:24.865055 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:35:24.865024 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8lcx\" (UniqueName: \"kubernetes.io/projected/3e0c7b26-181b-47f2-8503-b66092a86230-kube-api-access-r8lcx\") pod \"network-check-target-zsq7v\" (UID: \"3e0c7b26-181b-47f2-8503-b66092a86230\") " pod="openshift-network-diagnostics/network-check-target-zsq7v" Jul 07 14:35:24.867700 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:35:24.867681 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 14:35:24.877885 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:35:24.877866 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 14:35:24.888033 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:35:24.888008 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8lcx\" (UniqueName: \"kubernetes.io/projected/3e0c7b26-181b-47f2-8503-b66092a86230-kube-api-access-r8lcx\") pod \"network-check-target-zsq7v\" (UID: \"3e0c7b26-181b-47f2-8503-b66092a86230\") " pod="openshift-network-diagnostics/network-check-target-zsq7v" Jul 07 14:35:24.959880 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:35:24.959857 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dhggt\"" Jul 07 14:35:24.962828 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:35:24.962812 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bq6hn" Jul 07 14:35:24.967381 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:35:24.967361 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zsq7v" Jul 07 14:35:25.108094 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:35:25.108065 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bq6hn"] Jul 07 14:35:25.112406 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:35:25.112380 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod686ecb3c_7c9a_4ac1_ac27_4430651b0870.slice/crio-9c6de2d995cef32701cbdf60dc13cc6840a4bc886fd5829b46ebfaf1062e8e12 WatchSource:0}: Error finding container 9c6de2d995cef32701cbdf60dc13cc6840a4bc886fd5829b46ebfaf1062e8e12: Status 404 returned error can't find the container with id 9c6de2d995cef32701cbdf60dc13cc6840a4bc886fd5829b46ebfaf1062e8e12 Jul 07 14:35:25.125574 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:35:25.125551 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zsq7v"] Jul 07 14:35:25.128202 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:35:25.128181 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e0c7b26_181b_47f2_8503_b66092a86230.slice/crio-4ba31ab6cab5693f2435c2d2ebe0162de61be12641c73519d17365dae0bd125b WatchSource:0}: Error finding container 4ba31ab6cab5693f2435c2d2ebe0162de61be12641c73519d17365dae0bd125b: Status 404 returned error can't find the container with id 4ba31ab6cab5693f2435c2d2ebe0162de61be12641c73519d17365dae0bd125b Jul 07 14:35:25.332922 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:35:25.332880 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zsq7v" event={"ID":"3e0c7b26-181b-47f2-8503-b66092a86230","Type":"ContainerStarted","Data":"4ba31ab6cab5693f2435c2d2ebe0162de61be12641c73519d17365dae0bd125b"} Jul 07 14:35:25.333888 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:35:25.333855 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bq6hn" event={"ID":"686ecb3c-7c9a-4ac1-ac27-4430651b0870","Type":"ContainerStarted","Data":"9c6de2d995cef32701cbdf60dc13cc6840a4bc886fd5829b46ebfaf1062e8e12"} Jul 07 14:35:25.870861 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:35:25.870822 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c447425e-fb20-4ac8-b7e7-ed07d60953e5-metrics-tls\") pod \"dns-default-9pd7h\" (UID: \"c447425e-fb20-4ac8-b7e7-ed07d60953e5\") " pod="openshift-dns/dns-default-9pd7h" Jul 07 14:35:25.871288 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:35:25.871010 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 14:35:25.871288 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:35:25.871090 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c447425e-fb20-4ac8-b7e7-ed07d60953e5-metrics-tls podName:c447425e-fb20-4ac8-b7e7-ed07d60953e5 nodeName:}" failed. No retries permitted until 2026-07-07 14:35:57.87106818 +0000 UTC m=+99.295254649 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c447425e-fb20-4ac8-b7e7-ed07d60953e5-metrics-tls") pod "dns-default-9pd7h" (UID: "c447425e-fb20-4ac8-b7e7-ed07d60953e5") : secret "dns-default-metrics-tls" not found Jul 07 14:35:30.349217 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:35:30.349179 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zsq7v" event={"ID":"3e0c7b26-181b-47f2-8503-b66092a86230","Type":"ContainerStarted","Data":"992d8d7f76c0ccb327c4be305ec92a6a007667224899b6d567bf6133a9dab263"} Jul 07 14:35:30.349726 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:35:30.349306 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zsq7v" Jul 07 14:35:30.352861 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:35:30.352835 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bq6hn" event={"ID":"686ecb3c-7c9a-4ac1-ac27-4430651b0870","Type":"ContainerStarted","Data":"6165760f95dbe14bc9313865692ea07de0186e734c9a465f93aa5ff12802da04"} Jul 07 14:35:30.371891 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:35:30.371833 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zsq7v" podStartSLOduration=65.314406269 podStartE2EDuration="1m10.371816452s" totalImagesPullingTime="5.057410183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:34:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:35:30.371609009 +0000 UTC m=+71.795795484" watchObservedRunningTime="2026-07-07 14:35:30.371816452 +0000 UTC m=+71.796002928" Jul 07 14:35:30.391053 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:35:30.391007 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-bq6hn" podStartSLOduration=65.318756942 podStartE2EDuration="1m10.390989459s" totalImagesPullingTime="5.072232517s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:34:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:35:30.389881355 +0000 UTC m=+71.814067829" watchObservedRunningTime="2026-07-07 14:35:30.390989459 +0000 UTC m=+71.815175935" Jul 07 14:35:57.887238 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:35:57.887097 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c447425e-fb20-4ac8-b7e7-ed07d60953e5-metrics-tls\") pod \"dns-default-9pd7h\" (UID: \"c447425e-fb20-4ac8-b7e7-ed07d60953e5\") " pod="openshift-dns/dns-default-9pd7h" Jul 07 14:35:57.887238 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:35:57.887234 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 14:35:57.887747 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:35:57.887301 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c447425e-fb20-4ac8-b7e7-ed07d60953e5-metrics-tls podName:c447425e-fb20-4ac8-b7e7-ed07d60953e5 nodeName:}" failed. No retries permitted until 2026-07-07 14:37:01.88728192 +0000 UTC m=+163.311468373 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c447425e-fb20-4ac8-b7e7-ed07d60953e5-metrics-tls") pod "dns-default-9pd7h" (UID: "c447425e-fb20-4ac8-b7e7-ed07d60953e5") : secret "dns-default-metrics-tls" not found Jul 07 14:36:01.357398 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:36:01.357365 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zsq7v" Jul 07 14:36:26.809298 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:36:26.809257 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-546b8bc964-fx87n"] Jul 07 14:36:26.811594 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:36:26.811569 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-546b8bc964-fx87n" Jul 07 14:36:26.814071 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:36:26.814054 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-kl4wg\"" Jul 07 14:36:26.815166 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:36:26.815151 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 07 14:36:26.815265 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:36:26.815188 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 07 14:36:26.815328 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:36:26.815278 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 07 14:36:26.815382 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:36:26.815329 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 07 14:36:26.819334 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:36:26.819316 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-546b8bc964-fx87n"] Jul 07 14:36:26.975758 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:36:26.975719 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0443c633-667b-43cc-8960-8c35dfa69251-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-546b8bc964-fx87n\" (UID: \"0443c633-667b-43cc-8960-8c35dfa69251\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-546b8bc964-fx87n" Jul 07 14:36:26.975936 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:36:26.975828 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-455tf\" (UniqueName: \"kubernetes.io/projected/0443c633-667b-43cc-8960-8c35dfa69251-kube-api-access-455tf\") pod \"managed-serviceaccount-addon-agent-546b8bc964-fx87n\" (UID: \"0443c633-667b-43cc-8960-8c35dfa69251\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-546b8bc964-fx87n" Jul 07 14:36:27.076572 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:36:27.076498 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-455tf\" (UniqueName: \"kubernetes.io/projected/0443c633-667b-43cc-8960-8c35dfa69251-kube-api-access-455tf\") pod \"managed-serviceaccount-addon-agent-546b8bc964-fx87n\" (UID: \"0443c633-667b-43cc-8960-8c35dfa69251\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-546b8bc964-fx87n" Jul 07 14:36:27.076572 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:36:27.076551 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0443c633-667b-43cc-8960-8c35dfa69251-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-546b8bc964-fx87n\" (UID: \"0443c633-667b-43cc-8960-8c35dfa69251\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-546b8bc964-fx87n" Jul 07 14:36:27.079042 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:36:27.079024 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0443c633-667b-43cc-8960-8c35dfa69251-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-546b8bc964-fx87n\" (UID: \"0443c633-667b-43cc-8960-8c35dfa69251\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-546b8bc964-fx87n" Jul 07 14:36:27.085463 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:36:27.085438 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-455tf\" (UniqueName: \"kubernetes.io/projected/0443c633-667b-43cc-8960-8c35dfa69251-kube-api-access-455tf\") pod \"managed-serviceaccount-addon-agent-546b8bc964-fx87n\" (UID: \"0443c633-667b-43cc-8960-8c35dfa69251\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-546b8bc964-fx87n" Jul 07 14:36:27.131440 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:36:27.131407 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-546b8bc964-fx87n" Jul 07 14:36:27.252244 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:36:27.252213 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-546b8bc964-fx87n"] Jul 07 14:36:27.256266 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:36:27.256240 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0443c633_667b_43cc_8960_8c35dfa69251.slice/crio-991e74d37e1aab35b3df628754a41dca5916e6de450aa95d02528c50b575f86c WatchSource:0}: Error finding container 991e74d37e1aab35b3df628754a41dca5916e6de450aa95d02528c50b575f86c: Status 404 returned error can't find the container with id 991e74d37e1aab35b3df628754a41dca5916e6de450aa95d02528c50b575f86c Jul 07 14:36:27.490693 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:36:27.490662 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-546b8bc964-fx87n" event={"ID":"0443c633-667b-43cc-8960-8c35dfa69251","Type":"ContainerStarted","Data":"991e74d37e1aab35b3df628754a41dca5916e6de450aa95d02528c50b575f86c"} Jul 07 14:36:28.789361 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:36:28.789322 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3bbd20c-73f5-4d24-aed9-145e4b89a590-metrics-certs\") pod \"network-metrics-daemon-lpmtl\" (UID: \"e3bbd20c-73f5-4d24-aed9-145e4b89a590\") " pod="openshift-multus/network-metrics-daemon-lpmtl" Jul 07 14:36:28.789828 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:36:28.789471 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 07 14:36:28.789828 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:36:28.789543 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3bbd20c-73f5-4d24-aed9-145e4b89a590-metrics-certs podName:e3bbd20c-73f5-4d24-aed9-145e4b89a590 nodeName:}" failed. No retries permitted until 2026-07-07 14:38:30.789521112 +0000 UTC m=+252.213707564 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3bbd20c-73f5-4d24-aed9-145e4b89a590-metrics-certs") pod "network-metrics-daemon-lpmtl" (UID: "e3bbd20c-73f5-4d24-aed9-145e4b89a590") : secret "metrics-daemon-secret" not found Jul 07 14:36:30.500067 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:36:30.500030 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-546b8bc964-fx87n" event={"ID":"0443c633-667b-43cc-8960-8c35dfa69251","Type":"ContainerStarted","Data":"3804a1c1520a31b08eabf0ea0d9e39e533852a22ed2d8b3dbd58e92c37249b94"} Jul 07 14:36:56.999337 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:36:56.999281 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-9pd7h" podUID="c447425e-fb20-4ac8-b7e7-ed07d60953e5" Jul 07 14:36:57.048986 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:36:57.048954 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-lpmtl" podUID="e3bbd20c-73f5-4d24-aed9-145e4b89a590" Jul 07 14:36:57.563556 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:36:57.563528 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9pd7h" Jul 07 14:37:01.909871 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:37:01.909811 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c447425e-fb20-4ac8-b7e7-ed07d60953e5-metrics-tls\") pod \"dns-default-9pd7h\" (UID: \"c447425e-fb20-4ac8-b7e7-ed07d60953e5\") " pod="openshift-dns/dns-default-9pd7h" Jul 07 14:37:01.910284 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:37:01.909961 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 14:37:01.910284 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:37:01.910028 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c447425e-fb20-4ac8-b7e7-ed07d60953e5-metrics-tls podName:c447425e-fb20-4ac8-b7e7-ed07d60953e5 nodeName:}" failed. No retries permitted until 2026-07-07 14:39:03.910011822 +0000 UTC m=+285.334198275 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c447425e-fb20-4ac8-b7e7-ed07d60953e5-metrics-tls") pod "dns-default-9pd7h" (UID: "c447425e-fb20-4ac8-b7e7-ed07d60953e5") : secret "dns-default-metrics-tls" not found Jul 07 14:37:10.036087 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:37:10.036039 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpmtl" Jul 07 14:38:30.785003 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:30.784965 2555 generic.go:356] "Generic (PLEG): container finished" podID="0443c633-667b-43cc-8960-8c35dfa69251" containerID="3804a1c1520a31b08eabf0ea0d9e39e533852a22ed2d8b3dbd58e92c37249b94" exitCode=255 Jul 07 14:38:30.785430 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:30.785036 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-546b8bc964-fx87n" event={"ID":"0443c633-667b-43cc-8960-8c35dfa69251","Type":"ContainerDied","Data":"3804a1c1520a31b08eabf0ea0d9e39e533852a22ed2d8b3dbd58e92c37249b94"} Jul 07 14:38:30.790406 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:30.790388 2555 scope.go:117] "RemoveContainer" containerID="3804a1c1520a31b08eabf0ea0d9e39e533852a22ed2d8b3dbd58e92c37249b94" Jul 07 14:38:30.795546 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:30.795526 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3bbd20c-73f5-4d24-aed9-145e4b89a590-metrics-certs\") pod \"network-metrics-daemon-lpmtl\" (UID: \"e3bbd20c-73f5-4d24-aed9-145e4b89a590\") " pod="openshift-multus/network-metrics-daemon-lpmtl" Jul 07 14:38:30.795668 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:30.795653 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 07 14:38:30.795730 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:30.795719 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3bbd20c-73f5-4d24-aed9-145e4b89a590-metrics-certs podName:e3bbd20c-73f5-4d24-aed9-145e4b89a590 nodeName:}" failed. No retries permitted until 2026-07-07 14:40:32.795696956 +0000 UTC m=+374.219883410 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3bbd20c-73f5-4d24-aed9-145e4b89a590-metrics-certs") pod "network-metrics-daemon-lpmtl" (UID: "e3bbd20c-73f5-4d24-aed9-145e4b89a590") : secret "metrics-daemon-secret" not found Jul 07 14:38:31.789412 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:31.789375 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-546b8bc964-fx87n" event={"ID":"0443c633-667b-43cc-8960-8c35dfa69251","Type":"ContainerStarted","Data":"d49d826351171e771e81ee868c9944363c178e6a27ee47331ec2d36c69b2a2b7"} Jul 07 14:38:31.807000 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:31.806957 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-546b8bc964-fx87n" podStartSLOduration=123.08104173 podStartE2EDuration="2m5.806942855s" totalImagesPullingTime="2.725901125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:36:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:36:30.516931315 +0000 UTC m=+131.941117790" watchObservedRunningTime="2026-07-07 14:38:31.806942855 +0000 UTC m=+253.231129330" Jul 07 14:38:34.373268 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.373234 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-b56rb"] Jul 07 14:38:34.374989 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.374970 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-vxc5l"] Jul 07 14:38:34.375103 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.375085 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-b56rb" Jul 07 14:38:34.376824 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.376799 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-vxc5l" Jul 07 14:38:34.376995 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.376975 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-r9jmn"] Jul 07 14:38:34.377265 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.377246 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 07 14:38:34.377515 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.377499 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 07 14:38:34.377607 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.377588 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-5wdsw\"" Jul 07 14:38:34.377981 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.377959 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 07 14:38:34.377981 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.377973 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 07 14:38:34.378622 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.378607 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-8578b59545-z44lc"] Jul 07 14:38:34.378707 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.378695 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-r9jmn" Jul 07 14:38:34.379150 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.379135 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 07 14:38:34.379237 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.379138 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 07 14:38:34.380357 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.380338 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8578b59545-z44lc" Jul 07 14:38:34.380439 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.380426 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-xjhs2\"" Jul 07 14:38:34.380998 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.380968 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 07 14:38:34.381286 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.381272 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-r86dg\"" Jul 07 14:38:34.382935 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.382917 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 07 14:38:34.383065 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.383050 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 07 14:38:34.383137 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.383127 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-hkx8c\"" Jul 07 14:38:34.383578 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.383564 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 07 14:38:34.383765 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.383740 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 07 14:38:34.383909 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.383895 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 07 14:38:34.384042 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.384028 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 07 14:38:34.384598 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.384584 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 07 14:38:34.397584 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.397561 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-vxc5l"] Jul 07 14:38:34.398448 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.398426 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-r9jmn"] Jul 07 14:38:34.399268 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.399252 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-b56rb"] Jul 07 14:38:34.400146 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.400128 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-8578b59545-z44lc"] Jul 07 14:38:34.474059 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.474028 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-4zhnv"] Jul 07 14:38:34.476068 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.476041 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-4zhnv" Jul 07 14:38:34.476791 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.476769 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-4jw8p"] Jul 07 14:38:34.478834 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.478818 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-98gk6"] Jul 07 14:38:34.478963 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.478928 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-4jw8p" Jul 07 14:38:34.479652 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.479633 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 07 14:38:34.480688 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.480671 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6d8677489b-52crb"] Jul 07 14:38:34.480818 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.480803 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-98gk6" Jul 07 14:38:34.481224 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.481201 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 07 14:38:34.481395 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.481375 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 07 14:38:34.482532 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.482515 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d8677489b-52crb" Jul 07 14:38:34.482613 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.482598 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 07 14:38:34.482708 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.482685 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-bvpgx\"" Jul 07 14:38:34.484933 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.484918 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 07 14:38:34.485182 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.485167 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 07 14:38:34.485270 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.485210 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-s96tw\"" Jul 07 14:38:34.485327 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.485297 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 07 14:38:34.485598 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.485585 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 07 14:38:34.485790 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.485774 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 07 14:38:34.485837 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.485814 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-49bkn\"" Jul 07 14:38:34.487069 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.487055 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 07 14:38:34.487239 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.487226 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tnl6d\"" Jul 07 14:38:34.487387 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.487376 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 07 14:38:34.488445 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.488432 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 07 14:38:34.494835 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.494816 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 07 14:38:34.499415 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.499385 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-4zhnv"] Jul 07 14:38:34.508542 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.508500 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-4jw8p"] Jul 07 14:38:34.509696 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.509680 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-98gk6"] Jul 07 14:38:34.510820 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.510799 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d8677489b-52crb"] Jul 07 14:38:34.519334 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.519316 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fcd37998-008a-4bf0-b960-bbc6db0dbe76-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-vxc5l\" (UID: \"fcd37998-008a-4bf0-b960-bbc6db0dbe76\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-vxc5l" Jul 07 14:38:34.519426 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.519353 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk6vx\" (UniqueName: \"kubernetes.io/projected/76dd393c-d59d-4978-83f6-7bcdab3169c8-kube-api-access-lk6vx\") pod \"router-default-8578b59545-z44lc\" (UID: \"76dd393c-d59d-4978-83f6-7bcdab3169c8\") " pod="openshift-ingress/router-default-8578b59545-z44lc" Jul 07 14:38:34.519426 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.519373 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/75aeac0c-a9eb-44a9-9ed1-d7412dab8785-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-b56rb\" (UID: \"75aeac0c-a9eb-44a9-9ed1-d7412dab8785\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-b56rb" Jul 07 14:38:34.519426 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.519392 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sz55\" (UniqueName: \"kubernetes.io/projected/fcd37998-008a-4bf0-b960-bbc6db0dbe76-kube-api-access-6sz55\") pod \"service-ca-operator-5d46d9ffd4-vxc5l\" (UID: \"fcd37998-008a-4bf0-b960-bbc6db0dbe76\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-vxc5l" Jul 07 14:38:34.519541 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.519432 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76dd393c-d59d-4978-83f6-7bcdab3169c8-service-ca-bundle\") pod \"router-default-8578b59545-z44lc\" (UID: \"76dd393c-d59d-4978-83f6-7bcdab3169c8\") " pod="openshift-ingress/router-default-8578b59545-z44lc" Jul 07 14:38:34.519541 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.519510 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fcd37998-008a-4bf0-b960-bbc6db0dbe76-config\") pod \"service-ca-operator-5d46d9ffd4-vxc5l\" (UID: \"fcd37998-008a-4bf0-b960-bbc6db0dbe76\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-vxc5l" Jul 07 14:38:34.519541 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.519527 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76dd393c-d59d-4978-83f6-7bcdab3169c8-metrics-certs\") pod \"router-default-8578b59545-z44lc\" (UID: \"76dd393c-d59d-4978-83f6-7bcdab3169c8\") " pod="openshift-ingress/router-default-8578b59545-z44lc" Jul 07 14:38:34.519638 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.519543 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/75aeac0c-a9eb-44a9-9ed1-d7412dab8785-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-b56rb\" (UID: \"75aeac0c-a9eb-44a9-9ed1-d7412dab8785\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-b56rb" Jul 07 14:38:34.519638 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.519561 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/76dd393c-d59d-4978-83f6-7bcdab3169c8-default-certificate\") pod \"router-default-8578b59545-z44lc\" (UID: \"76dd393c-d59d-4978-83f6-7bcdab3169c8\") " pod="openshift-ingress/router-default-8578b59545-z44lc" Jul 07 14:38:34.519638 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.519580 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/76dd393c-d59d-4978-83f6-7bcdab3169c8-stats-auth\") pod \"router-default-8578b59545-z44lc\" (UID: \"76dd393c-d59d-4978-83f6-7bcdab3169c8\") " pod="openshift-ingress/router-default-8578b59545-z44lc" Jul 07 14:38:34.519638 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.519618 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qx6z\" (UniqueName: \"kubernetes.io/projected/75aeac0c-a9eb-44a9-9ed1-d7412dab8785-kube-api-access-8qx6z\") pod \"kube-storage-version-migrator-operator-846bf668f7-b56rb\" (UID: \"75aeac0c-a9eb-44a9-9ed1-d7412dab8785\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-b56rb" Jul 07 14:38:34.519776 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.519652 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7mtr\" (UniqueName: \"kubernetes.io/projected/a9ea50d7-08a3-4732-9ff6-09e16bcb31c9-kube-api-access-h7mtr\") pod \"network-check-source-989f8b8dc-r9jmn\" (UID: \"a9ea50d7-08a3-4732-9ff6-09e16bcb31c9\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-r9jmn" Jul 07 14:38:34.620197 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.620166 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/76dd393c-d59d-4978-83f6-7bcdab3169c8-default-certificate\") pod \"router-default-8578b59545-z44lc\" (UID: \"76dd393c-d59d-4978-83f6-7bcdab3169c8\") " pod="openshift-ingress/router-default-8578b59545-z44lc" Jul 07 14:38:34.620420 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.620203 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/76dd393c-d59d-4978-83f6-7bcdab3169c8-stats-auth\") pod \"router-default-8578b59545-z44lc\" (UID: \"76dd393c-d59d-4978-83f6-7bcdab3169c8\") " pod="openshift-ingress/router-default-8578b59545-z44lc" Jul 07 14:38:34.620420 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.620230 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxm4c\" (UniqueName: \"kubernetes.io/projected/a237e875-6fcb-4b05-bfff-6742c35b7afa-kube-api-access-qxm4c\") pod \"cluster-monitoring-operator-64786d9cbb-4zhnv\" (UID: \"a237e875-6fcb-4b05-bfff-6742c35b7afa\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-4zhnv" Jul 07 14:38:34.620420 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.620256 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/18ec5f4a-e826-4534-a068-3fb383136ca1-ca-trust-extracted\") pod \"image-registry-6d8677489b-52crb\" (UID: \"18ec5f4a-e826-4534-a068-3fb383136ca1\") " pod="openshift-image-registry/image-registry-6d8677489b-52crb" Jul 07 14:38:34.620420 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.620335 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxmsn\" (UniqueName: \"kubernetes.io/projected/f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc-kube-api-access-fxmsn\") pod \"cluster-samples-operator-56fcf86bf-98gk6\" (UID: \"f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-98gk6" Jul 07 14:38:34.620420 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.620377 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a237e875-6fcb-4b05-bfff-6742c35b7afa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-4zhnv\" (UID: \"a237e875-6fcb-4b05-bfff-6742c35b7afa\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-4zhnv" Jul 07 14:38:34.620639 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.620421 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fcd37998-008a-4bf0-b960-bbc6db0dbe76-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-vxc5l\" (UID: \"fcd37998-008a-4bf0-b960-bbc6db0dbe76\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-vxc5l" Jul 07 14:38:34.620639 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.620445 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxs9z\" (UniqueName: \"kubernetes.io/projected/a6580919-d925-4965-ba84-e035fa29ad2a-kube-api-access-zxs9z\") pod \"volume-data-source-validator-5d456fdbcd-4jw8p\" (UID: \"a6580919-d925-4965-ba84-e035fa29ad2a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-4jw8p" Jul 07 14:38:34.620639 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.620482 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdtp2\" (UniqueName: \"kubernetes.io/projected/18ec5f4a-e826-4534-a068-3fb383136ca1-kube-api-access-mdtp2\") pod \"image-registry-6d8677489b-52crb\" (UID: \"18ec5f4a-e826-4534-a068-3fb383136ca1\") " pod="openshift-image-registry/image-registry-6d8677489b-52crb" Jul 07 14:38:34.620639 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.620509 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sz55\" (UniqueName: \"kubernetes.io/projected/fcd37998-008a-4bf0-b960-bbc6db0dbe76-kube-api-access-6sz55\") pod \"service-ca-operator-5d46d9ffd4-vxc5l\" (UID: \"fcd37998-008a-4bf0-b960-bbc6db0dbe76\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-vxc5l" Jul 07 14:38:34.620639 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.620533 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76dd393c-d59d-4978-83f6-7bcdab3169c8-service-ca-bundle\") pod \"router-default-8578b59545-z44lc\" (UID: \"76dd393c-d59d-4978-83f6-7bcdab3169c8\") " pod="openshift-ingress/router-default-8578b59545-z44lc" Jul 07 14:38:34.620639 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.620557 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/18ec5f4a-e826-4534-a068-3fb383136ca1-image-registry-private-configuration\") pod \"image-registry-6d8677489b-52crb\" (UID: \"18ec5f4a-e826-4534-a068-3fb383136ca1\") " pod="openshift-image-registry/image-registry-6d8677489b-52crb" Jul 07 14:38:34.620639 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.620592 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/18ec5f4a-e826-4534-a068-3fb383136ca1-trusted-ca\") pod \"image-registry-6d8677489b-52crb\" (UID: \"18ec5f4a-e826-4534-a068-3fb383136ca1\") " pod="openshift-image-registry/image-registry-6d8677489b-52crb" Jul 07 14:38:34.620986 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.620653 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/18ec5f4a-e826-4534-a068-3fb383136ca1-installation-pull-secrets\") pod \"image-registry-6d8677489b-52crb\" (UID: \"18ec5f4a-e826-4534-a068-3fb383136ca1\") " pod="openshift-image-registry/image-registry-6d8677489b-52crb" Jul 07 14:38:34.620986 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:34.620737 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76dd393c-d59d-4978-83f6-7bcdab3169c8-service-ca-bundle podName:76dd393c-d59d-4978-83f6-7bcdab3169c8 nodeName:}" failed. No retries permitted until 2026-07-07 14:38:35.120700573 +0000 UTC m=+256.544887046 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/76dd393c-d59d-4978-83f6-7bcdab3169c8-service-ca-bundle") pod "router-default-8578b59545-z44lc" (UID: "76dd393c-d59d-4978-83f6-7bcdab3169c8") : configmap references non-existent config key: service-ca.crt Jul 07 14:38:34.620986 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.620816 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qx6z\" (UniqueName: \"kubernetes.io/projected/75aeac0c-a9eb-44a9-9ed1-d7412dab8785-kube-api-access-8qx6z\") pod \"kube-storage-version-migrator-operator-846bf668f7-b56rb\" (UID: \"75aeac0c-a9eb-44a9-9ed1-d7412dab8785\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-b56rb" Jul 07 14:38:34.620986 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.620855 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-98gk6\" (UID: \"f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-98gk6" Jul 07 14:38:34.620986 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.620889 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7mtr\" (UniqueName: \"kubernetes.io/projected/a9ea50d7-08a3-4732-9ff6-09e16bcb31c9-kube-api-access-h7mtr\") pod \"network-check-source-989f8b8dc-r9jmn\" (UID: \"a9ea50d7-08a3-4732-9ff6-09e16bcb31c9\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-r9jmn" Jul 07 14:38:34.620986 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.620931 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/18ec5f4a-e826-4534-a068-3fb383136ca1-bound-sa-token\") pod \"image-registry-6d8677489b-52crb\" (UID: \"18ec5f4a-e826-4534-a068-3fb383136ca1\") " pod="openshift-image-registry/image-registry-6d8677489b-52crb" Jul 07 14:38:34.620986 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.620971 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk6vx\" (UniqueName: \"kubernetes.io/projected/76dd393c-d59d-4978-83f6-7bcdab3169c8-kube-api-access-lk6vx\") pod \"router-default-8578b59545-z44lc\" (UID: \"76dd393c-d59d-4978-83f6-7bcdab3169c8\") " pod="openshift-ingress/router-default-8578b59545-z44lc" Jul 07 14:38:34.621285 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.621002 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/75aeac0c-a9eb-44a9-9ed1-d7412dab8785-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-b56rb\" (UID: \"75aeac0c-a9eb-44a9-9ed1-d7412dab8785\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-b56rb" Jul 07 14:38:34.621285 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.621039 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18ec5f4a-e826-4534-a068-3fb383136ca1-registry-tls\") pod \"image-registry-6d8677489b-52crb\" (UID: \"18ec5f4a-e826-4534-a068-3fb383136ca1\") " pod="openshift-image-registry/image-registry-6d8677489b-52crb" Jul 07 14:38:34.621285 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.621067 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a237e875-6fcb-4b05-bfff-6742c35b7afa-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-4zhnv\" (UID: \"a237e875-6fcb-4b05-bfff-6742c35b7afa\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-4zhnv" Jul 07 14:38:34.621285 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.621131 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/18ec5f4a-e826-4534-a068-3fb383136ca1-registry-certificates\") pod \"image-registry-6d8677489b-52crb\" (UID: \"18ec5f4a-e826-4534-a068-3fb383136ca1\") " pod="openshift-image-registry/image-registry-6d8677489b-52crb" Jul 07 14:38:34.621285 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.621176 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76dd393c-d59d-4978-83f6-7bcdab3169c8-metrics-certs\") pod \"router-default-8578b59545-z44lc\" (UID: \"76dd393c-d59d-4978-83f6-7bcdab3169c8\") " pod="openshift-ingress/router-default-8578b59545-z44lc" Jul 07 14:38:34.621285 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.621209 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fcd37998-008a-4bf0-b960-bbc6db0dbe76-config\") pod \"service-ca-operator-5d46d9ffd4-vxc5l\" (UID: \"fcd37998-008a-4bf0-b960-bbc6db0dbe76\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-vxc5l" Jul 07 14:38:34.621285 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.621237 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/75aeac0c-a9eb-44a9-9ed1-d7412dab8785-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-b56rb\" (UID: \"75aeac0c-a9eb-44a9-9ed1-d7412dab8785\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-b56rb" Jul 07 14:38:34.621285 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:34.621262 2555 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 07 14:38:34.621555 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:34.621304 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76dd393c-d59d-4978-83f6-7bcdab3169c8-metrics-certs podName:76dd393c-d59d-4978-83f6-7bcdab3169c8 nodeName:}" failed. No retries permitted until 2026-07-07 14:38:35.121290091 +0000 UTC m=+256.545476553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76dd393c-d59d-4978-83f6-7bcdab3169c8-metrics-certs") pod "router-default-8578b59545-z44lc" (UID: "76dd393c-d59d-4978-83f6-7bcdab3169c8") : secret "router-metrics-certs-default" not found Jul 07 14:38:34.621648 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.621613 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/75aeac0c-a9eb-44a9-9ed1-d7412dab8785-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-b56rb\" (UID: \"75aeac0c-a9eb-44a9-9ed1-d7412dab8785\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-b56rb" Jul 07 14:38:34.621837 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.621820 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fcd37998-008a-4bf0-b960-bbc6db0dbe76-config\") pod \"service-ca-operator-5d46d9ffd4-vxc5l\" (UID: \"fcd37998-008a-4bf0-b960-bbc6db0dbe76\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-vxc5l" Jul 07 14:38:34.622975 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.622948 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/76dd393c-d59d-4978-83f6-7bcdab3169c8-default-certificate\") pod \"router-default-8578b59545-z44lc\" (UID: \"76dd393c-d59d-4978-83f6-7bcdab3169c8\") " pod="openshift-ingress/router-default-8578b59545-z44lc" Jul 07 14:38:34.623086 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.623036 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fcd37998-008a-4bf0-b960-bbc6db0dbe76-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-vxc5l\" (UID: \"fcd37998-008a-4bf0-b960-bbc6db0dbe76\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-vxc5l" Jul 07 14:38:34.623188 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.623170 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/76dd393c-d59d-4978-83f6-7bcdab3169c8-stats-auth\") pod \"router-default-8578b59545-z44lc\" (UID: \"76dd393c-d59d-4978-83f6-7bcdab3169c8\") " pod="openshift-ingress/router-default-8578b59545-z44lc" Jul 07 14:38:34.623372 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.623337 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/75aeac0c-a9eb-44a9-9ed1-d7412dab8785-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-b56rb\" (UID: \"75aeac0c-a9eb-44a9-9ed1-d7412dab8785\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-b56rb" Jul 07 14:38:34.631419 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.631397 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk6vx\" (UniqueName: \"kubernetes.io/projected/76dd393c-d59d-4978-83f6-7bcdab3169c8-kube-api-access-lk6vx\") pod \"router-default-8578b59545-z44lc\" (UID: \"76dd393c-d59d-4978-83f6-7bcdab3169c8\") " pod="openshift-ingress/router-default-8578b59545-z44lc" Jul 07 14:38:34.634221 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.634195 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sz55\" (UniqueName: \"kubernetes.io/projected/fcd37998-008a-4bf0-b960-bbc6db0dbe76-kube-api-access-6sz55\") pod \"service-ca-operator-5d46d9ffd4-vxc5l\" (UID: \"fcd37998-008a-4bf0-b960-bbc6db0dbe76\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-vxc5l" Jul 07 14:38:34.634290 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.634195 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qx6z\" (UniqueName: \"kubernetes.io/projected/75aeac0c-a9eb-44a9-9ed1-d7412dab8785-kube-api-access-8qx6z\") pod \"kube-storage-version-migrator-operator-846bf668f7-b56rb\" (UID: \"75aeac0c-a9eb-44a9-9ed1-d7412dab8785\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-b56rb" Jul 07 14:38:34.634338 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.634292 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7mtr\" (UniqueName: \"kubernetes.io/projected/a9ea50d7-08a3-4732-9ff6-09e16bcb31c9-kube-api-access-h7mtr\") pod \"network-check-source-989f8b8dc-r9jmn\" (UID: \"a9ea50d7-08a3-4732-9ff6-09e16bcb31c9\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-r9jmn" Jul 07 14:38:34.685897 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.685874 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-b56rb" Jul 07 14:38:34.691450 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.691435 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-vxc5l" Jul 07 14:38:34.698132 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.698112 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-r9jmn" Jul 07 14:38:34.722231 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.722185 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18ec5f4a-e826-4534-a068-3fb383136ca1-registry-tls\") pod \"image-registry-6d8677489b-52crb\" (UID: \"18ec5f4a-e826-4534-a068-3fb383136ca1\") " pod="openshift-image-registry/image-registry-6d8677489b-52crb" Jul 07 14:38:34.722376 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.722238 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a237e875-6fcb-4b05-bfff-6742c35b7afa-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-4zhnv\" (UID: \"a237e875-6fcb-4b05-bfff-6742c35b7afa\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-4zhnv" Jul 07 14:38:34.722376 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.722264 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/18ec5f4a-e826-4534-a068-3fb383136ca1-registry-certificates\") pod \"image-registry-6d8677489b-52crb\" (UID: \"18ec5f4a-e826-4534-a068-3fb383136ca1\") " pod="openshift-image-registry/image-registry-6d8677489b-52crb" Jul 07 14:38:34.722376 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.722319 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxm4c\" (UniqueName: \"kubernetes.io/projected/a237e875-6fcb-4b05-bfff-6742c35b7afa-kube-api-access-qxm4c\") pod \"cluster-monitoring-operator-64786d9cbb-4zhnv\" (UID: \"a237e875-6fcb-4b05-bfff-6742c35b7afa\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-4zhnv" Jul 07 14:38:34.722376 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.722346 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/18ec5f4a-e826-4534-a068-3fb383136ca1-ca-trust-extracted\") pod \"image-registry-6d8677489b-52crb\" (UID: \"18ec5f4a-e826-4534-a068-3fb383136ca1\") " pod="openshift-image-registry/image-registry-6d8677489b-52crb" Jul 07 14:38:34.722585 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:34.722372 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 14:38:34.722585 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.722392 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxmsn\" (UniqueName: \"kubernetes.io/projected/f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc-kube-api-access-fxmsn\") pod \"cluster-samples-operator-56fcf86bf-98gk6\" (UID: \"f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-98gk6" Jul 07 14:38:34.722585 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.722419 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a237e875-6fcb-4b05-bfff-6742c35b7afa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-4zhnv\" (UID: \"a237e875-6fcb-4b05-bfff-6742c35b7afa\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-4zhnv" Jul 07 14:38:34.722585 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.722450 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxs9z\" (UniqueName: \"kubernetes.io/projected/a6580919-d925-4965-ba84-e035fa29ad2a-kube-api-access-zxs9z\") pod \"volume-data-source-validator-5d456fdbcd-4jw8p\" (UID: \"a6580919-d925-4965-ba84-e035fa29ad2a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-4jw8p" Jul 07 14:38:34.722585 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.722491 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdtp2\" (UniqueName: \"kubernetes.io/projected/18ec5f4a-e826-4534-a068-3fb383136ca1-kube-api-access-mdtp2\") pod \"image-registry-6d8677489b-52crb\" (UID: \"18ec5f4a-e826-4534-a068-3fb383136ca1\") " pod="openshift-image-registry/image-registry-6d8677489b-52crb" Jul 07 14:38:34.722585 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.722530 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/18ec5f4a-e826-4534-a068-3fb383136ca1-image-registry-private-configuration\") pod \"image-registry-6d8677489b-52crb\" (UID: \"18ec5f4a-e826-4534-a068-3fb383136ca1\") " pod="openshift-image-registry/image-registry-6d8677489b-52crb" Jul 07 14:38:34.722585 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.722567 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/18ec5f4a-e826-4534-a068-3fb383136ca1-trusted-ca\") pod \"image-registry-6d8677489b-52crb\" (UID: \"18ec5f4a-e826-4534-a068-3fb383136ca1\") " pod="openshift-image-registry/image-registry-6d8677489b-52crb" Jul 07 14:38:34.722941 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.722591 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/18ec5f4a-e826-4534-a068-3fb383136ca1-installation-pull-secrets\") pod \"image-registry-6d8677489b-52crb\" (UID: \"18ec5f4a-e826-4534-a068-3fb383136ca1\") " pod="openshift-image-registry/image-registry-6d8677489b-52crb" Jul 07 14:38:34.722941 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.722623 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-98gk6\" (UID: \"f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-98gk6" Jul 07 14:38:34.722941 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.722690 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/18ec5f4a-e826-4534-a068-3fb383136ca1-bound-sa-token\") pod \"image-registry-6d8677489b-52crb\" (UID: \"18ec5f4a-e826-4534-a068-3fb383136ca1\") " pod="openshift-image-registry/image-registry-6d8677489b-52crb" Jul 07 14:38:34.723093 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.723035 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/18ec5f4a-e826-4534-a068-3fb383136ca1-registry-certificates\") pod \"image-registry-6d8677489b-52crb\" (UID: \"18ec5f4a-e826-4534-a068-3fb383136ca1\") " pod="openshift-image-registry/image-registry-6d8677489b-52crb" Jul 07 14:38:34.723093 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:34.722397 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d8677489b-52crb: secret "image-registry-tls" not found Jul 07 14:38:34.723189 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:34.723139 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18ec5f4a-e826-4534-a068-3fb383136ca1-registry-tls podName:18ec5f4a-e826-4534-a068-3fb383136ca1 nodeName:}" failed. No retries permitted until 2026-07-07 14:38:35.223116977 +0000 UTC m=+256.647303446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/18ec5f4a-e826-4534-a068-3fb383136ca1-registry-tls") pod "image-registry-6d8677489b-52crb" (UID: "18ec5f4a-e826-4534-a068-3fb383136ca1") : secret "image-registry-tls" not found Jul 07 14:38:34.723512 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.723237 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/18ec5f4a-e826-4534-a068-3fb383136ca1-ca-trust-extracted\") pod \"image-registry-6d8677489b-52crb\" (UID: \"18ec5f4a-e826-4534-a068-3fb383136ca1\") " pod="openshift-image-registry/image-registry-6d8677489b-52crb" Jul 07 14:38:34.723512 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.723266 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a237e875-6fcb-4b05-bfff-6742c35b7afa-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-4zhnv\" (UID: \"a237e875-6fcb-4b05-bfff-6742c35b7afa\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-4zhnv" Jul 07 14:38:34.723512 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:34.723352 2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 14:38:34.723512 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:34.723435 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a237e875-6fcb-4b05-bfff-6742c35b7afa-cluster-monitoring-operator-tls podName:a237e875-6fcb-4b05-bfff-6742c35b7afa nodeName:}" failed. No retries permitted until 2026-07-07 14:38:35.223416983 +0000 UTC m=+256.647603436 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a237e875-6fcb-4b05-bfff-6742c35b7afa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-4zhnv" (UID: "a237e875-6fcb-4b05-bfff-6742c35b7afa") : secret "cluster-monitoring-operator-tls" not found Jul 07 14:38:34.723512 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:34.723505 2555 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 14:38:34.723868 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:34.723547 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc-samples-operator-tls podName:f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc nodeName:}" failed. No retries permitted until 2026-07-07 14:38:35.223533217 +0000 UTC m=+256.647719671 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-98gk6" (UID: "f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc") : secret "samples-operator-tls" not found Jul 07 14:38:34.724155 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.724111 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/18ec5f4a-e826-4534-a068-3fb383136ca1-trusted-ca\") pod \"image-registry-6d8677489b-52crb\" (UID: \"18ec5f4a-e826-4534-a068-3fb383136ca1\") " pod="openshift-image-registry/image-registry-6d8677489b-52crb" Jul 07 14:38:34.727607 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.727583 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/18ec5f4a-e826-4534-a068-3fb383136ca1-image-registry-private-configuration\") pod \"image-registry-6d8677489b-52crb\" (UID: \"18ec5f4a-e826-4534-a068-3fb383136ca1\") " pod="openshift-image-registry/image-registry-6d8677489b-52crb" Jul 07 14:38:34.728050 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.728029 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/18ec5f4a-e826-4534-a068-3fb383136ca1-installation-pull-secrets\") pod \"image-registry-6d8677489b-52crb\" (UID: \"18ec5f4a-e826-4534-a068-3fb383136ca1\") " pod="openshift-image-registry/image-registry-6d8677489b-52crb" Jul 07 14:38:34.735131 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.734957 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/18ec5f4a-e826-4534-a068-3fb383136ca1-bound-sa-token\") pod \"image-registry-6d8677489b-52crb\" (UID: \"18ec5f4a-e826-4534-a068-3fb383136ca1\") " pod="openshift-image-registry/image-registry-6d8677489b-52crb" Jul 07 14:38:34.735131 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.735065 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxmsn\" (UniqueName: \"kubernetes.io/projected/f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc-kube-api-access-fxmsn\") pod \"cluster-samples-operator-56fcf86bf-98gk6\" (UID: \"f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-98gk6" Jul 07 14:38:34.735333 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.735311 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxs9z\" (UniqueName: \"kubernetes.io/projected/a6580919-d925-4965-ba84-e035fa29ad2a-kube-api-access-zxs9z\") pod \"volume-data-source-validator-5d456fdbcd-4jw8p\" (UID: \"a6580919-d925-4965-ba84-e035fa29ad2a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-4jw8p" Jul 07 14:38:34.735390 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.735376 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxm4c\" (UniqueName: \"kubernetes.io/projected/a237e875-6fcb-4b05-bfff-6742c35b7afa-kube-api-access-qxm4c\") pod \"cluster-monitoring-operator-64786d9cbb-4zhnv\" (UID: \"a237e875-6fcb-4b05-bfff-6742c35b7afa\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-4zhnv" Jul 07 14:38:34.735626 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.735607 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdtp2\" (UniqueName: \"kubernetes.io/projected/18ec5f4a-e826-4534-a068-3fb383136ca1-kube-api-access-mdtp2\") pod \"image-registry-6d8677489b-52crb\" (UID: \"18ec5f4a-e826-4534-a068-3fb383136ca1\") " pod="openshift-image-registry/image-registry-6d8677489b-52crb" Jul 07 14:38:34.793408 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.793385 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-4jw8p" Jul 07 14:38:34.829481 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.829444 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-b56rb"] Jul 07 14:38:34.832448 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:38:34.832413 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75aeac0c_a9eb_44a9_9ed1_d7412dab8785.slice/crio-4767210b4a32d760d65b9c5d755a4714cdd439db58a5692961dee55345f33491 WatchSource:0}: Error finding container 4767210b4a32d760d65b9c5d755a4714cdd439db58a5692961dee55345f33491: Status 404 returned error can't find the container with id 4767210b4a32d760d65b9c5d755a4714cdd439db58a5692961dee55345f33491 Jul 07 14:38:34.922175 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:34.922132 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-4jw8p"] Jul 07 14:38:34.925668 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:38:34.925645 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6580919_d925_4965_ba84_e035fa29ad2a.slice/crio-6899ca65c0dbf98bd10f92f48ec21d9ca7c6e3748d3638d896860cafbd7f5db6 WatchSource:0}: Error finding container 6899ca65c0dbf98bd10f92f48ec21d9ca7c6e3748d3638d896860cafbd7f5db6: Status 404 returned error can't find the container with id 6899ca65c0dbf98bd10f92f48ec21d9ca7c6e3748d3638d896860cafbd7f5db6 Jul 07 14:38:35.055117 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:35.055090 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-r9jmn"] Jul 07 14:38:35.057358 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:38:35.057327 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9ea50d7_08a3_4732_9ff6_09e16bcb31c9.slice/crio-59bff66015cffcfa686321f4b9947a90b3540d17b2a294a5d09cb62381a47996 WatchSource:0}: Error finding container 59bff66015cffcfa686321f4b9947a90b3540d17b2a294a5d09cb62381a47996: Status 404 returned error can't find the container with id 59bff66015cffcfa686321f4b9947a90b3540d17b2a294a5d09cb62381a47996 Jul 07 14:38:35.057679 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:35.057579 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-vxc5l"] Jul 07 14:38:35.059768 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:38:35.059726 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcd37998_008a_4bf0_b960_bbc6db0dbe76.slice/crio-78a7634a2e4f0ccccbecba3cb61aba5c2f83256c730a19f41a95ca63c8e0ef2d WatchSource:0}: Error finding container 78a7634a2e4f0ccccbecba3cb61aba5c2f83256c730a19f41a95ca63c8e0ef2d: Status 404 returned error can't find the container with id 78a7634a2e4f0ccccbecba3cb61aba5c2f83256c730a19f41a95ca63c8e0ef2d Jul 07 14:38:35.127607 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:35.127540 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76dd393c-d59d-4978-83f6-7bcdab3169c8-metrics-certs\") pod \"router-default-8578b59545-z44lc\" (UID: \"76dd393c-d59d-4978-83f6-7bcdab3169c8\") " pod="openshift-ingress/router-default-8578b59545-z44lc" Jul 07 14:38:35.127738 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:35.127621 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76dd393c-d59d-4978-83f6-7bcdab3169c8-service-ca-bundle\") pod \"router-default-8578b59545-z44lc\" (UID: \"76dd393c-d59d-4978-83f6-7bcdab3169c8\") " pod="openshift-ingress/router-default-8578b59545-z44lc" Jul 07 14:38:35.127738 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:35.127690 2555 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 07 14:38:35.127885 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:35.127773 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76dd393c-d59d-4978-83f6-7bcdab3169c8-metrics-certs podName:76dd393c-d59d-4978-83f6-7bcdab3169c8 nodeName:}" failed. No retries permitted until 2026-07-07 14:38:36.127738915 +0000 UTC m=+257.551925386 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76dd393c-d59d-4978-83f6-7bcdab3169c8-metrics-certs") pod "router-default-8578b59545-z44lc" (UID: "76dd393c-d59d-4978-83f6-7bcdab3169c8") : secret "router-metrics-certs-default" not found Jul 07 14:38:35.127885 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:35.127855 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76dd393c-d59d-4978-83f6-7bcdab3169c8-service-ca-bundle podName:76dd393c-d59d-4978-83f6-7bcdab3169c8 nodeName:}" failed. No retries permitted until 2026-07-07 14:38:36.127835196 +0000 UTC m=+257.552021663 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/76dd393c-d59d-4978-83f6-7bcdab3169c8-service-ca-bundle") pod "router-default-8578b59545-z44lc" (UID: "76dd393c-d59d-4978-83f6-7bcdab3169c8") : configmap references non-existent config key: service-ca.crt Jul 07 14:38:35.229101 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:35.229065 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18ec5f4a-e826-4534-a068-3fb383136ca1-registry-tls\") pod \"image-registry-6d8677489b-52crb\" (UID: \"18ec5f4a-e826-4534-a068-3fb383136ca1\") " pod="openshift-image-registry/image-registry-6d8677489b-52crb" Jul 07 14:38:35.229273 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:35.229217 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 14:38:35.229273 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:35.229233 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a237e875-6fcb-4b05-bfff-6742c35b7afa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-4zhnv\" (UID: \"a237e875-6fcb-4b05-bfff-6742c35b7afa\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-4zhnv" Jul 07 14:38:35.229378 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:35.229240 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d8677489b-52crb: secret "image-registry-tls" not found Jul 07 14:38:35.229378 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:35.229301 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-98gk6\" (UID: \"f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-98gk6" Jul 07 14:38:35.229378 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:35.229356 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18ec5f4a-e826-4534-a068-3fb383136ca1-registry-tls podName:18ec5f4a-e826-4534-a068-3fb383136ca1 nodeName:}" failed. No retries permitted until 2026-07-07 14:38:36.229336266 +0000 UTC m=+257.653522738 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/18ec5f4a-e826-4534-a068-3fb383136ca1-registry-tls") pod "image-registry-6d8677489b-52crb" (UID: "18ec5f4a-e826-4534-a068-3fb383136ca1") : secret "image-registry-tls" not found Jul 07 14:38:35.229535 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:35.229292 2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 14:38:35.229535 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:35.229425 2555 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 14:38:35.229535 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:35.229465 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a237e875-6fcb-4b05-bfff-6742c35b7afa-cluster-monitoring-operator-tls podName:a237e875-6fcb-4b05-bfff-6742c35b7afa nodeName:}" failed. No retries permitted until 2026-07-07 14:38:36.229445493 +0000 UTC m=+257.653631952 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a237e875-6fcb-4b05-bfff-6742c35b7afa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-4zhnv" (UID: "a237e875-6fcb-4b05-bfff-6742c35b7afa") : secret "cluster-monitoring-operator-tls" not found Jul 07 14:38:35.229535 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:35.229486 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc-samples-operator-tls podName:f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc nodeName:}" failed. No retries permitted until 2026-07-07 14:38:36.229474648 +0000 UTC m=+257.653661102 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-98gk6" (UID: "f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc") : secret "samples-operator-tls" not found Jul 07 14:38:35.805401 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:35.805361 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-r9jmn" event={"ID":"a9ea50d7-08a3-4732-9ff6-09e16bcb31c9","Type":"ContainerStarted","Data":"38e7a1d16112c674dbcc7469719ce6b7524fbb5ec27c53a9aba56d617293ec56"} Jul 07 14:38:35.805870 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:35.805422 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-r9jmn" event={"ID":"a9ea50d7-08a3-4732-9ff6-09e16bcb31c9","Type":"ContainerStarted","Data":"59bff66015cffcfa686321f4b9947a90b3540d17b2a294a5d09cb62381a47996"} Jul 07 14:38:35.808942 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:35.808846 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-b56rb" event={"ID":"75aeac0c-a9eb-44a9-9ed1-d7412dab8785","Type":"ContainerStarted","Data":"4767210b4a32d760d65b9c5d755a4714cdd439db58a5692961dee55345f33491"} Jul 07 14:38:35.811308 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:35.811240 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-4jw8p" event={"ID":"a6580919-d925-4965-ba84-e035fa29ad2a","Type":"ContainerStarted","Data":"6899ca65c0dbf98bd10f92f48ec21d9ca7c6e3748d3638d896860cafbd7f5db6"} Jul 07 14:38:35.812957 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:35.812902 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-vxc5l" event={"ID":"fcd37998-008a-4bf0-b960-bbc6db0dbe76","Type":"ContainerStarted","Data":"78a7634a2e4f0ccccbecba3cb61aba5c2f83256c730a19f41a95ca63c8e0ef2d"} Jul 07 14:38:35.823244 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:35.823193 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-r9jmn" podStartSLOduration=1.823174568 podStartE2EDuration="1.823174568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:38:35.822488588 +0000 UTC m=+257.246675064" watchObservedRunningTime="2026-07-07 14:38:35.823174568 +0000 UTC m=+257.247361044" Jul 07 14:38:36.136867 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:36.136838 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76dd393c-d59d-4978-83f6-7bcdab3169c8-service-ca-bundle\") pod \"router-default-8578b59545-z44lc\" (UID: \"76dd393c-d59d-4978-83f6-7bcdab3169c8\") " pod="openshift-ingress/router-default-8578b59545-z44lc" Jul 07 14:38:36.136984 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:36.136946 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76dd393c-d59d-4978-83f6-7bcdab3169c8-metrics-certs\") pod \"router-default-8578b59545-z44lc\" (UID: \"76dd393c-d59d-4978-83f6-7bcdab3169c8\") " pod="openshift-ingress/router-default-8578b59545-z44lc" Jul 07 14:38:36.137041 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:36.136992 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76dd393c-d59d-4978-83f6-7bcdab3169c8-service-ca-bundle podName:76dd393c-d59d-4978-83f6-7bcdab3169c8 nodeName:}" failed. No retries permitted until 2026-07-07 14:38:38.136972803 +0000 UTC m=+259.561159260 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/76dd393c-d59d-4978-83f6-7bcdab3169c8-service-ca-bundle") pod "router-default-8578b59545-z44lc" (UID: "76dd393c-d59d-4978-83f6-7bcdab3169c8") : configmap references non-existent config key: service-ca.crt Jul 07 14:38:36.137109 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:36.137060 2555 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 07 14:38:36.137166 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:36.137109 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76dd393c-d59d-4978-83f6-7bcdab3169c8-metrics-certs podName:76dd393c-d59d-4978-83f6-7bcdab3169c8 nodeName:}" failed. No retries permitted until 2026-07-07 14:38:38.137094036 +0000 UTC m=+259.561280489 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76dd393c-d59d-4978-83f6-7bcdab3169c8-metrics-certs") pod "router-default-8578b59545-z44lc" (UID: "76dd393c-d59d-4978-83f6-7bcdab3169c8") : secret "router-metrics-certs-default" not found Jul 07 14:38:36.238325 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:36.238292 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a237e875-6fcb-4b05-bfff-6742c35b7afa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-4zhnv\" (UID: \"a237e875-6fcb-4b05-bfff-6742c35b7afa\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-4zhnv" Jul 07 14:38:36.238465 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:36.238376 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-98gk6\" (UID: \"f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-98gk6" Jul 07 14:38:36.238465 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:36.238422 2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 14:38:36.238465 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:36.238438 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18ec5f4a-e826-4534-a068-3fb383136ca1-registry-tls\") pod \"image-registry-6d8677489b-52crb\" (UID: \"18ec5f4a-e826-4534-a068-3fb383136ca1\") " pod="openshift-image-registry/image-registry-6d8677489b-52crb" Jul 07 14:38:36.238557 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:36.238485 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a237e875-6fcb-4b05-bfff-6742c35b7afa-cluster-monitoring-operator-tls podName:a237e875-6fcb-4b05-bfff-6742c35b7afa nodeName:}" failed. No retries permitted until 2026-07-07 14:38:38.238467215 +0000 UTC m=+259.662653681 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a237e875-6fcb-4b05-bfff-6742c35b7afa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-4zhnv" (UID: "a237e875-6fcb-4b05-bfff-6742c35b7afa") : secret "cluster-monitoring-operator-tls" not found Jul 07 14:38:36.238557 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:36.238531 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 14:38:36.238557 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:36.238543 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d8677489b-52crb: secret "image-registry-tls" not found Jul 07 14:38:36.238654 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:36.238578 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18ec5f4a-e826-4534-a068-3fb383136ca1-registry-tls podName:18ec5f4a-e826-4534-a068-3fb383136ca1 nodeName:}" failed. No retries permitted until 2026-07-07 14:38:38.238566817 +0000 UTC m=+259.662753286 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/18ec5f4a-e826-4534-a068-3fb383136ca1-registry-tls") pod "image-registry-6d8677489b-52crb" (UID: "18ec5f4a-e826-4534-a068-3fb383136ca1") : secret "image-registry-tls" not found Jul 07 14:38:36.238654 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:36.238629 2555 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 14:38:36.238716 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:36.238657 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc-samples-operator-tls podName:f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc nodeName:}" failed. No retries permitted until 2026-07-07 14:38:38.238647582 +0000 UTC m=+259.662834049 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-98gk6" (UID: "f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc") : secret "samples-operator-tls" not found Jul 07 14:38:36.818011 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:36.817966 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-4jw8p" event={"ID":"a6580919-d925-4965-ba84-e035fa29ad2a","Type":"ContainerStarted","Data":"1b0fbef391953b1c01c8f0a3c73aec6ca27d7b882ef998c13c7548ee5b9c0133"} Jul 07 14:38:36.836361 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:36.836313 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-4jw8p" podStartSLOduration=1.6593431870000002 podStartE2EDuration="2.836296291s" totalImagesPullingTime="1.176953104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:38:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:38:36.835418128 +0000 UTC m=+258.259604604" watchObservedRunningTime="2026-07-07 14:38:36.836296291 +0000 UTC m=+258.260482745" Jul 07 14:38:37.823146 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:37.823105 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-b56rb" event={"ID":"75aeac0c-a9eb-44a9-9ed1-d7412dab8785","Type":"ContainerStarted","Data":"3f6694e89484774b99a984c2cf89a764717566a1812b6f26fcc09edb2e755af1"} Jul 07 14:38:37.824667 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:37.824637 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-vxc5l" event={"ID":"fcd37998-008a-4bf0-b960-bbc6db0dbe76","Type":"ContainerStarted","Data":"8c2061de71408b78ad0e3a1f958137ede085fe02b954a50d1eb85676ea159e24"} Jul 07 14:38:37.841302 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:37.841250 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-b56rb" podStartSLOduration=1.01511072 podStartE2EDuration="3.841231878s" totalImagesPullingTime="2.826121158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:38:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:38:37.839708071 +0000 UTC m=+259.263894546" watchObservedRunningTime="2026-07-07 14:38:37.841231878 +0000 UTC m=+259.265418358" Jul 07 14:38:37.855598 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:37.855558 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-vxc5l" podStartSLOduration=1.254774674 podStartE2EDuration="3.855545108s" totalImagesPullingTime="2.600770434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:38:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:38:37.855373989 +0000 UTC m=+259.279560487" watchObservedRunningTime="2026-07-07 14:38:37.855545108 +0000 UTC m=+259.279731561" Jul 07 14:38:38.156524 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:38.156484 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76dd393c-d59d-4978-83f6-7bcdab3169c8-metrics-certs\") pod \"router-default-8578b59545-z44lc\" (UID: \"76dd393c-d59d-4978-83f6-7bcdab3169c8\") " pod="openshift-ingress/router-default-8578b59545-z44lc" Jul 07 14:38:38.156694 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:38.156577 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76dd393c-d59d-4978-83f6-7bcdab3169c8-service-ca-bundle\") pod \"router-default-8578b59545-z44lc\" (UID: \"76dd393c-d59d-4978-83f6-7bcdab3169c8\") " pod="openshift-ingress/router-default-8578b59545-z44lc" Jul 07 14:38:38.156694 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:38.156639 2555 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 07 14:38:38.156785 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:38.156702 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76dd393c-d59d-4978-83f6-7bcdab3169c8-metrics-certs podName:76dd393c-d59d-4978-83f6-7bcdab3169c8 nodeName:}" failed. No retries permitted until 2026-07-07 14:38:42.156686768 +0000 UTC m=+263.580873224 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76dd393c-d59d-4978-83f6-7bcdab3169c8-metrics-certs") pod "router-default-8578b59545-z44lc" (UID: "76dd393c-d59d-4978-83f6-7bcdab3169c8") : secret "router-metrics-certs-default" not found Jul 07 14:38:38.156827 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:38.156781 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76dd393c-d59d-4978-83f6-7bcdab3169c8-service-ca-bundle podName:76dd393c-d59d-4978-83f6-7bcdab3169c8 nodeName:}" failed. No retries permitted until 2026-07-07 14:38:42.156746736 +0000 UTC m=+263.580933196 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/76dd393c-d59d-4978-83f6-7bcdab3169c8-service-ca-bundle") pod "router-default-8578b59545-z44lc" (UID: "76dd393c-d59d-4978-83f6-7bcdab3169c8") : configmap references non-existent config key: service-ca.crt Jul 07 14:38:38.258024 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:38.257980 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18ec5f4a-e826-4534-a068-3fb383136ca1-registry-tls\") pod \"image-registry-6d8677489b-52crb\" (UID: \"18ec5f4a-e826-4534-a068-3fb383136ca1\") " pod="openshift-image-registry/image-registry-6d8677489b-52crb" Jul 07 14:38:38.258195 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:38.258081 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a237e875-6fcb-4b05-bfff-6742c35b7afa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-4zhnv\" (UID: \"a237e875-6fcb-4b05-bfff-6742c35b7afa\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-4zhnv" Jul 07 14:38:38.258195 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:38.258159 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-98gk6\" (UID: \"f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-98gk6" Jul 07 14:38:38.258195 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:38.258160 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 14:38:38.258300 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:38.258206 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d8677489b-52crb: secret "image-registry-tls" not found Jul 07 14:38:38.258300 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:38.258207 2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 14:38:38.258300 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:38.258267 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a237e875-6fcb-4b05-bfff-6742c35b7afa-cluster-monitoring-operator-tls podName:a237e875-6fcb-4b05-bfff-6742c35b7afa nodeName:}" failed. No retries permitted until 2026-07-07 14:38:42.258250968 +0000 UTC m=+263.682437423 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a237e875-6fcb-4b05-bfff-6742c35b7afa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-4zhnv" (UID: "a237e875-6fcb-4b05-bfff-6742c35b7afa") : secret "cluster-monitoring-operator-tls" not found Jul 07 14:38:38.258300 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:38.258278 2555 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 14:38:38.258443 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:38.258281 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18ec5f4a-e826-4534-a068-3fb383136ca1-registry-tls podName:18ec5f4a-e826-4534-a068-3fb383136ca1 nodeName:}" failed. No retries permitted until 2026-07-07 14:38:42.258275005 +0000 UTC m=+263.682461458 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/18ec5f4a-e826-4534-a068-3fb383136ca1-registry-tls") pod "image-registry-6d8677489b-52crb" (UID: "18ec5f4a-e826-4534-a068-3fb383136ca1") : secret "image-registry-tls" not found Jul 07 14:38:38.258443 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:38.258343 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc-samples-operator-tls podName:f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc nodeName:}" failed. No retries permitted until 2026-07-07 14:38:42.258326231 +0000 UTC m=+263.682512689 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-98gk6" (UID: "f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc") : secret "samples-operator-tls" not found Jul 07 14:38:38.380885 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:38.380848 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mpm2g"] Jul 07 14:38:38.383386 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:38.383361 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mpm2g" Jul 07 14:38:38.388351 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:38.388328 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 07 14:38:38.389267 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:38.389250 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 07 14:38:38.389800 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:38.389573 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 07 14:38:38.389800 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:38.389599 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-s99qs\"" Jul 07 14:38:38.395194 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:38.395174 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 07 14:38:38.402418 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:38.402399 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mpm2g"] Jul 07 14:38:38.459411 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:38.459327 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mpm2g\" (UID: \"73b85bdb-2070-4cfa-a85c-82f2f2065d6c\") " pod="openshift-insights/insights-runtime-extractor-mpm2g" Jul 07 14:38:38.459411 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:38.459365 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-data-volume\") pod \"insights-runtime-extractor-mpm2g\" (UID: \"73b85bdb-2070-4cfa-a85c-82f2f2065d6c\") " pod="openshift-insights/insights-runtime-extractor-mpm2g" Jul 07 14:38:38.459411 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:38.459400 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-crio-socket\") pod \"insights-runtime-extractor-mpm2g\" (UID: \"73b85bdb-2070-4cfa-a85c-82f2f2065d6c\") " pod="openshift-insights/insights-runtime-extractor-mpm2g" Jul 07 14:38:38.459662 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:38.459432 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mpm2g\" (UID: \"73b85bdb-2070-4cfa-a85c-82f2f2065d6c\") " pod="openshift-insights/insights-runtime-extractor-mpm2g" Jul 07 14:38:38.459662 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:38.459502 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzg9f\" (UniqueName: \"kubernetes.io/projected/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-kube-api-access-kzg9f\") pod \"insights-runtime-extractor-mpm2g\" (UID: \"73b85bdb-2070-4cfa-a85c-82f2f2065d6c\") " pod="openshift-insights/insights-runtime-extractor-mpm2g" Jul 07 14:38:38.560649 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:38.560609 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mpm2g\" (UID: \"73b85bdb-2070-4cfa-a85c-82f2f2065d6c\") " pod="openshift-insights/insights-runtime-extractor-mpm2g" Jul 07 14:38:38.560649 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:38.560648 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-data-volume\") pod \"insights-runtime-extractor-mpm2g\" (UID: \"73b85bdb-2070-4cfa-a85c-82f2f2065d6c\") " pod="openshift-insights/insights-runtime-extractor-mpm2g" Jul 07 14:38:38.560923 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:38.560721 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-crio-socket\") pod \"insights-runtime-extractor-mpm2g\" (UID: \"73b85bdb-2070-4cfa-a85c-82f2f2065d6c\") " pod="openshift-insights/insights-runtime-extractor-mpm2g" Jul 07 14:38:38.560923 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:38.560793 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mpm2g\" (UID: \"73b85bdb-2070-4cfa-a85c-82f2f2065d6c\") " pod="openshift-insights/insights-runtime-extractor-mpm2g" Jul 07 14:38:38.560923 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:38.560847 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 14:38:38.560923 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:38.560851 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzg9f\" (UniqueName: \"kubernetes.io/projected/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-kube-api-access-kzg9f\") pod \"insights-runtime-extractor-mpm2g\" (UID: \"73b85bdb-2070-4cfa-a85c-82f2f2065d6c\") " pod="openshift-insights/insights-runtime-extractor-mpm2g" Jul 07 14:38:38.560923 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:38.560910 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-insights-runtime-extractor-tls podName:73b85bdb-2070-4cfa-a85c-82f2f2065d6c nodeName:}" failed. No retries permitted until 2026-07-07 14:38:39.060890888 +0000 UTC m=+260.485077348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mpm2g" (UID: "73b85bdb-2070-4cfa-a85c-82f2f2065d6c") : secret "insights-runtime-extractor-tls" not found Jul 07 14:38:38.560923 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:38.560911 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-crio-socket\") pod \"insights-runtime-extractor-mpm2g\" (UID: \"73b85bdb-2070-4cfa-a85c-82f2f2065d6c\") " pod="openshift-insights/insights-runtime-extractor-mpm2g" Jul 07 14:38:38.560923 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:38.560923 2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 14:38:38.561270 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:38.560989 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-kube-rbac-proxy-cm podName:73b85bdb-2070-4cfa-a85c-82f2f2065d6c nodeName:}" failed. No retries permitted until 2026-07-07 14:38:39.060971229 +0000 UTC m=+260.485157689 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mpm2g" (UID: "73b85bdb-2070-4cfa-a85c-82f2f2065d6c") : configmap "kube-rbac-proxy" not found Jul 07 14:38:38.561270 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:38.561047 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-data-volume\") pod \"insights-runtime-extractor-mpm2g\" (UID: \"73b85bdb-2070-4cfa-a85c-82f2f2065d6c\") " pod="openshift-insights/insights-runtime-extractor-mpm2g" Jul 07 14:38:38.574726 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:38.574706 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzg9f\" (UniqueName: \"kubernetes.io/projected/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-kube-api-access-kzg9f\") pod \"insights-runtime-extractor-mpm2g\" (UID: \"73b85bdb-2070-4cfa-a85c-82f2f2065d6c\") " pod="openshift-insights/insights-runtime-extractor-mpm2g" Jul 07 14:38:39.065721 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:39.065694 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mpm2g\" (UID: \"73b85bdb-2070-4cfa-a85c-82f2f2065d6c\") " pod="openshift-insights/insights-runtime-extractor-mpm2g" Jul 07 14:38:39.066103 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:39.065743 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mpm2g\" (UID: \"73b85bdb-2070-4cfa-a85c-82f2f2065d6c\") " pod="openshift-insights/insights-runtime-extractor-mpm2g" Jul 07 14:38:39.066103 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:39.065874 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 14:38:39.066103 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:39.065881 2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 14:38:39.066103 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:39.065937 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-insights-runtime-extractor-tls podName:73b85bdb-2070-4cfa-a85c-82f2f2065d6c nodeName:}" failed. No retries permitted until 2026-07-07 14:38:40.065922795 +0000 UTC m=+261.490109257 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mpm2g" (UID: "73b85bdb-2070-4cfa-a85c-82f2f2065d6c") : secret "insights-runtime-extractor-tls" not found Jul 07 14:38:39.066103 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:39.065951 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-kube-rbac-proxy-cm podName:73b85bdb-2070-4cfa-a85c-82f2f2065d6c nodeName:}" failed. No retries permitted until 2026-07-07 14:38:40.065945999 +0000 UTC m=+261.490132452 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mpm2g" (UID: "73b85bdb-2070-4cfa-a85c-82f2f2065d6c") : configmap "kube-rbac-proxy" not found Jul 07 14:38:40.072258 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:40.072225 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mpm2g\" (UID: \"73b85bdb-2070-4cfa-a85c-82f2f2065d6c\") " pod="openshift-insights/insights-runtime-extractor-mpm2g" Jul 07 14:38:40.072616 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:40.072330 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mpm2g\" (UID: \"73b85bdb-2070-4cfa-a85c-82f2f2065d6c\") " pod="openshift-insights/insights-runtime-extractor-mpm2g" Jul 07 14:38:40.072616 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:40.072345 2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 14:38:40.072616 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:40.072405 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-kube-rbac-proxy-cm podName:73b85bdb-2070-4cfa-a85c-82f2f2065d6c nodeName:}" failed. No retries permitted until 2026-07-07 14:38:42.072390224 +0000 UTC m=+263.496576677 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mpm2g" (UID: "73b85bdb-2070-4cfa-a85c-82f2f2065d6c") : configmap "kube-rbac-proxy" not found Jul 07 14:38:40.072616 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:40.072426 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 14:38:40.072616 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:40.072463 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-insights-runtime-extractor-tls podName:73b85bdb-2070-4cfa-a85c-82f2f2065d6c nodeName:}" failed. No retries permitted until 2026-07-07 14:38:42.0724531 +0000 UTC m=+263.496639553 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mpm2g" (UID: "73b85bdb-2070-4cfa-a85c-82f2f2065d6c") : secret "insights-runtime-extractor-tls" not found Jul 07 14:38:40.118195 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:40.118164 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-czppd_553f749a-649c-41c0-aa39-a0cb13725ca8/dns-node-resolver/0.log" Jul 07 14:38:41.515850 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:41.515780 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vkwws_7d94747d-5e8a-4eab-8234-969f7dfbaee5/node-ca/0.log" Jul 07 14:38:42.089633 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:42.089596 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mpm2g\" (UID: \"73b85bdb-2070-4cfa-a85c-82f2f2065d6c\") " pod="openshift-insights/insights-runtime-extractor-mpm2g" Jul 07 14:38:42.089819 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:42.089648 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mpm2g\" (UID: \"73b85bdb-2070-4cfa-a85c-82f2f2065d6c\") " pod="openshift-insights/insights-runtime-extractor-mpm2g" Jul 07 14:38:42.089819 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:42.089742 2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 14:38:42.089819 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:42.089778 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 14:38:42.089937 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:42.089827 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-kube-rbac-proxy-cm podName:73b85bdb-2070-4cfa-a85c-82f2f2065d6c nodeName:}" failed. No retries permitted until 2026-07-07 14:38:46.089809725 +0000 UTC m=+267.513996201 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mpm2g" (UID: "73b85bdb-2070-4cfa-a85c-82f2f2065d6c") : configmap "kube-rbac-proxy" not found Jul 07 14:38:42.089937 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:42.089841 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-insights-runtime-extractor-tls podName:73b85bdb-2070-4cfa-a85c-82f2f2065d6c nodeName:}" failed. No retries permitted until 2026-07-07 14:38:46.089835343 +0000 UTC m=+267.514021795 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mpm2g" (UID: "73b85bdb-2070-4cfa-a85c-82f2f2065d6c") : secret "insights-runtime-extractor-tls" not found Jul 07 14:38:42.190736 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:42.190703 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76dd393c-d59d-4978-83f6-7bcdab3169c8-service-ca-bundle\") pod \"router-default-8578b59545-z44lc\" (UID: \"76dd393c-d59d-4978-83f6-7bcdab3169c8\") " pod="openshift-ingress/router-default-8578b59545-z44lc" Jul 07 14:38:42.190898 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:42.190825 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76dd393c-d59d-4978-83f6-7bcdab3169c8-metrics-certs\") pod \"router-default-8578b59545-z44lc\" (UID: \"76dd393c-d59d-4978-83f6-7bcdab3169c8\") " pod="openshift-ingress/router-default-8578b59545-z44lc" Jul 07 14:38:42.190898 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:42.190861 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76dd393c-d59d-4978-83f6-7bcdab3169c8-service-ca-bundle podName:76dd393c-d59d-4978-83f6-7bcdab3169c8 nodeName:}" failed. No retries permitted until 2026-07-07 14:38:50.190845312 +0000 UTC m=+271.615031765 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/76dd393c-d59d-4978-83f6-7bcdab3169c8-service-ca-bundle") pod "router-default-8578b59545-z44lc" (UID: "76dd393c-d59d-4978-83f6-7bcdab3169c8") : configmap references non-existent config key: service-ca.crt Jul 07 14:38:42.190979 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:42.190921 2555 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 07 14:38:42.190979 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:42.190970 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76dd393c-d59d-4978-83f6-7bcdab3169c8-metrics-certs podName:76dd393c-d59d-4978-83f6-7bcdab3169c8 nodeName:}" failed. No retries permitted until 2026-07-07 14:38:50.190955131 +0000 UTC m=+271.615141601 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76dd393c-d59d-4978-83f6-7bcdab3169c8-metrics-certs") pod "router-default-8578b59545-z44lc" (UID: "76dd393c-d59d-4978-83f6-7bcdab3169c8") : secret "router-metrics-certs-default" not found Jul 07 14:38:42.291414 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:42.291380 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18ec5f4a-e826-4534-a068-3fb383136ca1-registry-tls\") pod \"image-registry-6d8677489b-52crb\" (UID: \"18ec5f4a-e826-4534-a068-3fb383136ca1\") " pod="openshift-image-registry/image-registry-6d8677489b-52crb" Jul 07 14:38:42.291580 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:42.291435 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a237e875-6fcb-4b05-bfff-6742c35b7afa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-4zhnv\" (UID: \"a237e875-6fcb-4b05-bfff-6742c35b7afa\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-4zhnv" Jul 07 14:38:42.291580 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:42.291487 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-98gk6\" (UID: \"f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-98gk6" Jul 07 14:38:42.291580 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:42.291526 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 14:38:42.291580 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:42.291548 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d8677489b-52crb: secret "image-registry-tls" not found Jul 07 14:38:42.291580 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:42.291567 2555 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 14:38:42.291580 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:42.291572 2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 14:38:42.291859 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:42.291605 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc-samples-operator-tls podName:f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc nodeName:}" failed. No retries permitted until 2026-07-07 14:38:50.291591873 +0000 UTC m=+271.715778326 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-98gk6" (UID: "f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc") : secret "samples-operator-tls" not found Jul 07 14:38:42.291859 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:42.291616 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18ec5f4a-e826-4534-a068-3fb383136ca1-registry-tls podName:18ec5f4a-e826-4534-a068-3fb383136ca1 nodeName:}" failed. No retries permitted until 2026-07-07 14:38:50.291610881 +0000 UTC m=+271.715797334 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/18ec5f4a-e826-4534-a068-3fb383136ca1-registry-tls") pod "image-registry-6d8677489b-52crb" (UID: "18ec5f4a-e826-4534-a068-3fb383136ca1") : secret "image-registry-tls" not found Jul 07 14:38:42.291859 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:42.291627 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a237e875-6fcb-4b05-bfff-6742c35b7afa-cluster-monitoring-operator-tls podName:a237e875-6fcb-4b05-bfff-6742c35b7afa nodeName:}" failed. No retries permitted until 2026-07-07 14:38:50.291620988 +0000 UTC m=+271.715807441 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a237e875-6fcb-4b05-bfff-6742c35b7afa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-4zhnv" (UID: "a237e875-6fcb-4b05-bfff-6742c35b7afa") : secret "cluster-monitoring-operator-tls" not found Jul 07 14:38:42.317555 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:42.317529 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xv4vf_8c25e009-ed68-418d-b5c1-02d8fa948f74/serve-healthcheck-canary/0.log" Jul 07 14:38:46.121089 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:46.121040 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mpm2g\" (UID: \"73b85bdb-2070-4cfa-a85c-82f2f2065d6c\") " pod="openshift-insights/insights-runtime-extractor-mpm2g" Jul 07 14:38:46.121089 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:46.121103 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mpm2g\" (UID: \"73b85bdb-2070-4cfa-a85c-82f2f2065d6c\") " pod="openshift-insights/insights-runtime-extractor-mpm2g" Jul 07 14:38:46.121559 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:46.121166 2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 14:38:46.121559 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:46.121202 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 14:38:46.121559 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:46.121223 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-kube-rbac-proxy-cm podName:73b85bdb-2070-4cfa-a85c-82f2f2065d6c nodeName:}" failed. No retries permitted until 2026-07-07 14:38:54.12120914 +0000 UTC m=+275.545395593 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mpm2g" (UID: "73b85bdb-2070-4cfa-a85c-82f2f2065d6c") : configmap "kube-rbac-proxy" not found Jul 07 14:38:46.121559 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:46.121265 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-insights-runtime-extractor-tls podName:73b85bdb-2070-4cfa-a85c-82f2f2065d6c nodeName:}" failed. No retries permitted until 2026-07-07 14:38:54.1212468 +0000 UTC m=+275.545433254 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mpm2g" (UID: "73b85bdb-2070-4cfa-a85c-82f2f2065d6c") : secret "insights-runtime-extractor-tls" not found Jul 07 14:38:50.258112 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:50.258081 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76dd393c-d59d-4978-83f6-7bcdab3169c8-metrics-certs\") pod \"router-default-8578b59545-z44lc\" (UID: \"76dd393c-d59d-4978-83f6-7bcdab3169c8\") " pod="openshift-ingress/router-default-8578b59545-z44lc" Jul 07 14:38:50.258520 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:50.258147 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76dd393c-d59d-4978-83f6-7bcdab3169c8-service-ca-bundle\") pod \"router-default-8578b59545-z44lc\" (UID: \"76dd393c-d59d-4978-83f6-7bcdab3169c8\") " pod="openshift-ingress/router-default-8578b59545-z44lc" Jul 07 14:38:50.258520 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:50.258252 2555 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 07 14:38:50.258520 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:50.258309 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76dd393c-d59d-4978-83f6-7bcdab3169c8-service-ca-bundle podName:76dd393c-d59d-4978-83f6-7bcdab3169c8 nodeName:}" failed. No retries permitted until 2026-07-07 14:39:06.258293097 +0000 UTC m=+287.682479554 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/76dd393c-d59d-4978-83f6-7bcdab3169c8-service-ca-bundle") pod "router-default-8578b59545-z44lc" (UID: "76dd393c-d59d-4978-83f6-7bcdab3169c8") : configmap references non-existent config key: service-ca.crt Jul 07 14:38:50.258520 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:50.258330 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76dd393c-d59d-4978-83f6-7bcdab3169c8-metrics-certs podName:76dd393c-d59d-4978-83f6-7bcdab3169c8 nodeName:}" failed. No retries permitted until 2026-07-07 14:39:06.258322278 +0000 UTC m=+287.682508731 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76dd393c-d59d-4978-83f6-7bcdab3169c8-metrics-certs") pod "router-default-8578b59545-z44lc" (UID: "76dd393c-d59d-4978-83f6-7bcdab3169c8") : secret "router-metrics-certs-default" not found Jul 07 14:38:50.359210 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:50.359183 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18ec5f4a-e826-4534-a068-3fb383136ca1-registry-tls\") pod \"image-registry-6d8677489b-52crb\" (UID: \"18ec5f4a-e826-4534-a068-3fb383136ca1\") " pod="openshift-image-registry/image-registry-6d8677489b-52crb" Jul 07 14:38:50.359354 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:50.359235 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a237e875-6fcb-4b05-bfff-6742c35b7afa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-4zhnv\" (UID: \"a237e875-6fcb-4b05-bfff-6742c35b7afa\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-4zhnv" Jul 07 14:38:50.359354 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:50.359285 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-98gk6\" (UID: \"f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-98gk6" Jul 07 14:38:50.359476 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:50.359375 2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 14:38:50.359476 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:50.359429 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a237e875-6fcb-4b05-bfff-6742c35b7afa-cluster-monitoring-operator-tls podName:a237e875-6fcb-4b05-bfff-6742c35b7afa nodeName:}" failed. No retries permitted until 2026-07-07 14:39:06.359415141 +0000 UTC m=+287.783601594 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a237e875-6fcb-4b05-bfff-6742c35b7afa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-4zhnv" (UID: "a237e875-6fcb-4b05-bfff-6742c35b7afa") : secret "cluster-monitoring-operator-tls" not found Jul 07 14:38:50.361549 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:50.361522 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-98gk6\" (UID: \"f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-98gk6" Jul 07 14:38:50.361641 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:50.361574 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18ec5f4a-e826-4534-a068-3fb383136ca1-registry-tls\") pod \"image-registry-6d8677489b-52crb\" (UID: \"18ec5f4a-e826-4534-a068-3fb383136ca1\") " pod="openshift-image-registry/image-registry-6d8677489b-52crb" Jul 07 14:38:50.399743 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:50.399719 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-98gk6" Jul 07 14:38:50.404758 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:50.404736 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d8677489b-52crb" Jul 07 14:38:50.531857 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:50.531781 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-98gk6"] Jul 07 14:38:50.547801 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:50.547780 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d8677489b-52crb"] Jul 07 14:38:50.555738 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:38:50.555712 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18ec5f4a_e826_4534_a068_3fb383136ca1.slice/crio-f8ec536cc95426a1dabbef2833b52842d265d90c442b5fccfbcbe13c4897a036 WatchSource:0}: Error finding container f8ec536cc95426a1dabbef2833b52842d265d90c442b5fccfbcbe13c4897a036: Status 404 returned error can't find the container with id f8ec536cc95426a1dabbef2833b52842d265d90c442b5fccfbcbe13c4897a036 Jul 07 14:38:50.869122 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:50.869079 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d8677489b-52crb" event={"ID":"18ec5f4a-e826-4534-a068-3fb383136ca1","Type":"ContainerStarted","Data":"0d8cba06bdc6cc565705081fae435e15c4ae75bfda5ef1b73349c3007edb338c"} Jul 07 14:38:50.869122 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:50.869126 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d8677489b-52crb" event={"ID":"18ec5f4a-e826-4534-a068-3fb383136ca1","Type":"ContainerStarted","Data":"f8ec536cc95426a1dabbef2833b52842d265d90c442b5fccfbcbe13c4897a036"} Jul 07 14:38:50.869349 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:50.869163 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6d8677489b-52crb" Jul 07 14:38:50.870239 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:50.870218 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-98gk6" event={"ID":"f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc","Type":"ContainerStarted","Data":"b6c34bbffc9ce10a45c926232b6b520308c69cff9f89e5eff36fa1c5cfaa0486"} Jul 07 14:38:50.889260 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:50.889216 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6d8677489b-52crb" podStartSLOduration=16.889199624 podStartE2EDuration="16.889199624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:38:50.887620726 +0000 UTC m=+272.311807202" watchObservedRunningTime="2026-07-07 14:38:50.889199624 +0000 UTC m=+272.313386099" Jul 07 14:38:52.879975 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:52.879934 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-98gk6" event={"ID":"f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc","Type":"ContainerStarted","Data":"8208d44a7a90435c0358ca3639630939bddb20f54255c015e073ab2e94b3682a"} Jul 07 14:38:52.880339 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:52.879991 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-98gk6" event={"ID":"f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc","Type":"ContainerStarted","Data":"8fa7307a2c80fc74393d6fc4622ae89414d1327a770654c9a8ac6ebfef6a3b2f"} Jul 07 14:38:52.899512 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:52.899460 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-98gk6" podStartSLOduration=17.166351726 podStartE2EDuration="18.899446399s" totalImagesPullingTime="1.733094673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:38:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:38:52.898214796 +0000 UTC m=+274.322401270" watchObservedRunningTime="2026-07-07 14:38:52.899446399 +0000 UTC m=+274.323632873" Jul 07 14:38:54.193951 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:54.193918 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mpm2g\" (UID: \"73b85bdb-2070-4cfa-a85c-82f2f2065d6c\") " pod="openshift-insights/insights-runtime-extractor-mpm2g" Jul 07 14:38:54.194302 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:54.194013 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mpm2g\" (UID: \"73b85bdb-2070-4cfa-a85c-82f2f2065d6c\") " pod="openshift-insights/insights-runtime-extractor-mpm2g" Jul 07 14:38:54.194302 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:54.194093 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 14:38:54.194302 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:38:54.194142 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-insights-runtime-extractor-tls podName:73b85bdb-2070-4cfa-a85c-82f2f2065d6c nodeName:}" failed. No retries permitted until 2026-07-07 14:39:10.194129927 +0000 UTC m=+291.618316381 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mpm2g" (UID: "73b85bdb-2070-4cfa-a85c-82f2f2065d6c") : secret "insights-runtime-extractor-tls" not found Jul 07 14:38:54.194447 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:38:54.194428 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mpm2g\" (UID: \"73b85bdb-2070-4cfa-a85c-82f2f2065d6c\") " pod="openshift-insights/insights-runtime-extractor-mpm2g" Jul 07 14:39:00.564367 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:39:00.564321 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-9pd7h" podUID="c447425e-fb20-4ac8-b7e7-ed07d60953e5" Jul 07 14:39:00.905325 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:00.905291 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9pd7h" Jul 07 14:39:03.677826 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:03.677786 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d8677489b-52crb"] Jul 07 14:39:03.966586 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:03.966480 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c447425e-fb20-4ac8-b7e7-ed07d60953e5-metrics-tls\") pod \"dns-default-9pd7h\" (UID: \"c447425e-fb20-4ac8-b7e7-ed07d60953e5\") " pod="openshift-dns/dns-default-9pd7h" Jul 07 14:39:03.968800 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:03.968775 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c447425e-fb20-4ac8-b7e7-ed07d60953e5-metrics-tls\") pod \"dns-default-9pd7h\" (UID: \"c447425e-fb20-4ac8-b7e7-ed07d60953e5\") " pod="openshift-dns/dns-default-9pd7h" Jul 07 14:39:04.208427 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:04.208394 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-v5tsw\"" Jul 07 14:39:04.216975 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:04.216902 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9pd7h" Jul 07 14:39:04.342430 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:04.342399 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9pd7h"] Jul 07 14:39:04.345223 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:39:04.345194 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc447425e_fb20_4ac8_b7e7_ed07d60953e5.slice/crio-fff55abd3e42b4bb5a7f6d9f97840016d2be97f0f20680bf3ec82efe22b36f10 WatchSource:0}: Error finding container fff55abd3e42b4bb5a7f6d9f97840016d2be97f0f20680bf3ec82efe22b36f10: Status 404 returned error can't find the container with id fff55abd3e42b4bb5a7f6d9f97840016d2be97f0f20680bf3ec82efe22b36f10 Jul 07 14:39:04.918746 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:04.918711 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9pd7h" event={"ID":"c447425e-fb20-4ac8-b7e7-ed07d60953e5","Type":"ContainerStarted","Data":"fff55abd3e42b4bb5a7f6d9f97840016d2be97f0f20680bf3ec82efe22b36f10"} Jul 07 14:39:06.288214 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:06.288178 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76dd393c-d59d-4978-83f6-7bcdab3169c8-metrics-certs\") pod \"router-default-8578b59545-z44lc\" (UID: \"76dd393c-d59d-4978-83f6-7bcdab3169c8\") " pod="openshift-ingress/router-default-8578b59545-z44lc" Jul 07 14:39:06.288653 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:06.288254 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76dd393c-d59d-4978-83f6-7bcdab3169c8-service-ca-bundle\") pod \"router-default-8578b59545-z44lc\" (UID: \"76dd393c-d59d-4978-83f6-7bcdab3169c8\") " pod="openshift-ingress/router-default-8578b59545-z44lc" Jul 07 14:39:06.288898 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:06.288879 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76dd393c-d59d-4978-83f6-7bcdab3169c8-service-ca-bundle\") pod \"router-default-8578b59545-z44lc\" (UID: \"76dd393c-d59d-4978-83f6-7bcdab3169c8\") " pod="openshift-ingress/router-default-8578b59545-z44lc" Jul 07 14:39:06.290902 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:06.290873 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76dd393c-d59d-4978-83f6-7bcdab3169c8-metrics-certs\") pod \"router-default-8578b59545-z44lc\" (UID: \"76dd393c-d59d-4978-83f6-7bcdab3169c8\") " pod="openshift-ingress/router-default-8578b59545-z44lc" Jul 07 14:39:06.389175 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:06.389131 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a237e875-6fcb-4b05-bfff-6742c35b7afa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-4zhnv\" (UID: \"a237e875-6fcb-4b05-bfff-6742c35b7afa\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-4zhnv" Jul 07 14:39:06.392110 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:06.392082 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a237e875-6fcb-4b05-bfff-6742c35b7afa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-4zhnv\" (UID: \"a237e875-6fcb-4b05-bfff-6742c35b7afa\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-4zhnv" Jul 07 14:39:06.504227 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:06.504197 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8578b59545-z44lc" Jul 07 14:39:06.585785 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:06.585743 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-4zhnv" Jul 07 14:39:06.682560 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:06.682489 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-8578b59545-z44lc"] Jul 07 14:39:06.686087 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:39:06.686056 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76dd393c_d59d_4978_83f6_7bcdab3169c8.slice/crio-45e42e6a3e9ea152db0706e726540366d55fb36c1c5b0e388516e0814ec9fd08 WatchSource:0}: Error finding container 45e42e6a3e9ea152db0706e726540366d55fb36c1c5b0e388516e0814ec9fd08: Status 404 returned error can't find the container with id 45e42e6a3e9ea152db0706e726540366d55fb36c1c5b0e388516e0814ec9fd08 Jul 07 14:39:06.758522 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:06.758030 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-4zhnv"] Jul 07 14:39:06.760872 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:39:06.760846 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda237e875_6fcb_4b05_bfff_6742c35b7afa.slice/crio-a68e0a2021f240992a423c975492378af2f01519585d55d42fc845c22bcd8c61 WatchSource:0}: Error finding container a68e0a2021f240992a423c975492378af2f01519585d55d42fc845c22bcd8c61: Status 404 returned error can't find the container with id a68e0a2021f240992a423c975492378af2f01519585d55d42fc845c22bcd8c61 Jul 07 14:39:06.927266 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:06.927227 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9pd7h" event={"ID":"c447425e-fb20-4ac8-b7e7-ed07d60953e5","Type":"ContainerStarted","Data":"00c141cf7f314325ba858b660a017616cc31612622848826b44becda621e11eb"} Jul 07 14:39:06.927459 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:06.927285 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9pd7h" event={"ID":"c447425e-fb20-4ac8-b7e7-ed07d60953e5","Type":"ContainerStarted","Data":"dfbd764ae5f2f5542642563d4c3e25106e340de42282ac33e5867f846a6f7e9c"} Jul 07 14:39:06.927459 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:06.927323 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9pd7h" Jul 07 14:39:06.928324 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:06.928300 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-4zhnv" event={"ID":"a237e875-6fcb-4b05-bfff-6742c35b7afa","Type":"ContainerStarted","Data":"a68e0a2021f240992a423c975492378af2f01519585d55d42fc845c22bcd8c61"} Jul 07 14:39:06.929529 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:06.929508 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-8578b59545-z44lc" event={"ID":"76dd393c-d59d-4978-83f6-7bcdab3169c8","Type":"ContainerStarted","Data":"18fbca0390f042dcef1b3172682462414c2df46e52f52a97ce70b6b69fed43b0"} Jul 07 14:39:06.929609 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:06.929540 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-8578b59545-z44lc" event={"ID":"76dd393c-d59d-4978-83f6-7bcdab3169c8","Type":"ContainerStarted","Data":"45e42e6a3e9ea152db0706e726540366d55fb36c1c5b0e388516e0814ec9fd08"} Jul 07 14:39:06.948657 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:06.948615 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9pd7h" podStartSLOduration=251.806507134 podStartE2EDuration="4m13.948601534s" totalImagesPullingTime="2.1420944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:34:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:39:06.948209951 +0000 UTC m=+288.372396426" watchObservedRunningTime="2026-07-07 14:39:06.948601534 +0000 UTC m=+288.372788009" Jul 07 14:39:06.973328 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:06.973283 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-8578b59545-z44lc" podStartSLOduration=32.97326955 podStartE2EDuration="32.97326955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:39:06.972535613 +0000 UTC m=+288.396722088" watchObservedRunningTime="2026-07-07 14:39:06.97326955 +0000 UTC m=+288.397456025" Jul 07 14:39:07.504615 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:07.504580 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-8578b59545-z44lc" Jul 07 14:39:07.507120 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:07.507096 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-8578b59545-z44lc" Jul 07 14:39:07.937115 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:07.937076 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-8578b59545-z44lc" Jul 07 14:39:07.938219 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:07.938196 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-8578b59545-z44lc" Jul 07 14:39:09.947429 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:09.947384 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-4zhnv" event={"ID":"a237e875-6fcb-4b05-bfff-6742c35b7afa","Type":"ContainerStarted","Data":"f1faacbcf57fba04855abec1a42b362dfe8a2ecbce7c983dd7b76ba729186c34"} Jul 07 14:39:09.965662 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:09.965618 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-4zhnv" podStartSLOduration=33.71090825 podStartE2EDuration="35.965604698s" totalImagesPullingTime="2.254696448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:38:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:39:09.965094675 +0000 UTC m=+291.389281150" watchObservedRunningTime="2026-07-07 14:39:09.965604698 +0000 UTC m=+291.389791172" Jul 07 14:39:10.220946 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:10.220836 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mpm2g\" (UID: \"73b85bdb-2070-4cfa-a85c-82f2f2065d6c\") " pod="openshift-insights/insights-runtime-extractor-mpm2g" Jul 07 14:39:10.223156 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:10.223135 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73b85bdb-2070-4cfa-a85c-82f2f2065d6c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mpm2g\" (UID: \"73b85bdb-2070-4cfa-a85c-82f2f2065d6c\") " pod="openshift-insights/insights-runtime-extractor-mpm2g" Jul 07 14:39:10.493303 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:10.493210 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mpm2g" Jul 07 14:39:10.623989 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:10.623956 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mpm2g"] Jul 07 14:39:10.626604 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:39:10.626576 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73b85bdb_2070_4cfa_a85c_82f2f2065d6c.slice/crio-ec3a94a53b6e69f978bceae8c71124bbbde2a4da8a800279c7a278f2da636799 WatchSource:0}: Error finding container ec3a94a53b6e69f978bceae8c71124bbbde2a4da8a800279c7a278f2da636799: Status 404 returned error can't find the container with id ec3a94a53b6e69f978bceae8c71124bbbde2a4da8a800279c7a278f2da636799 Jul 07 14:39:10.953839 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:10.953803 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mpm2g" event={"ID":"73b85bdb-2070-4cfa-a85c-82f2f2065d6c","Type":"ContainerStarted","Data":"ec3a94a53b6e69f978bceae8c71124bbbde2a4da8a800279c7a278f2da636799"} Jul 07 14:39:13.037104 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:39:13.037058 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-lpmtl" podUID="e3bbd20c-73f5-4d24-aed9-145e4b89a590" Jul 07 14:39:13.683761 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:13.683709 2555 patch_prober.go:28] interesting pod/image-registry-6d8677489b-52crb container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 07 14:39:13.683941 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:13.683797 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6d8677489b-52crb" podUID="18ec5f4a-e826-4534-a068-3fb383136ca1" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 14:39:14.970923 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:14.970890 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mpm2g" event={"ID":"73b85bdb-2070-4cfa-a85c-82f2f2065d6c","Type":"ContainerStarted","Data":"ba704101637478b99580a1f19189c74146da6f2146908acea84fa8d2060dccaa"} Jul 07 14:39:15.977090 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:15.977051 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mpm2g" event={"ID":"73b85bdb-2070-4cfa-a85c-82f2f2065d6c","Type":"ContainerStarted","Data":"f3427ab68a4e6a3532d5c16ddeb78482a3ce07a1f048fdcd600de8203feb8818"} Jul 07 14:39:16.939059 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:16.939027 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9pd7h" Jul 07 14:39:17.986736 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:17.986701 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mpm2g" event={"ID":"73b85bdb-2070-4cfa-a85c-82f2f2065d6c","Type":"ContainerStarted","Data":"0da0f297ecf7b307c826ee5f226ccb947408e7913cf74dc865ced33e00677a9a"} Jul 07 14:39:18.004660 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:18.004610 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mpm2g" podStartSLOduration=33.453373723 podStartE2EDuration="40.004594965s" totalImagesPullingTime="6.551221242s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:38:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:39:18.003943145 +0000 UTC m=+299.428129618" watchObservedRunningTime="2026-07-07 14:39:18.004594965 +0000 UTC m=+299.428781439" Jul 07 14:39:18.748327 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:18.748290 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8674d957bc-bhcjr"] Jul 07 14:39:18.750205 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:18.750189 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8674d957bc-bhcjr" Jul 07 14:39:18.754200 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:18.754160 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 07 14:39:18.754200 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:18.754163 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7svhg\"" Jul 07 14:39:18.754396 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:18.754231 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 07 14:39:18.754396 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:18.754260 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 07 14:39:18.754396 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:18.754320 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 07 14:39:18.754540 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:18.754458 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 07 14:39:18.754540 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:18.754495 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 07 14:39:18.754540 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:18.754513 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 07 14:39:18.758214 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:18.758195 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 07 14:39:18.763168 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:18.763148 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8674d957bc-bhcjr"] Jul 07 14:39:18.889048 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:18.889016 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/63f16300-0736-4975-ad35-2c6a5876338d-console-serving-cert\") pod \"console-8674d957bc-bhcjr\" (UID: \"63f16300-0736-4975-ad35-2c6a5876338d\") " pod="openshift-console/console-8674d957bc-bhcjr" Jul 07 14:39:18.889250 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:18.889068 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/63f16300-0736-4975-ad35-2c6a5876338d-console-config\") pod \"console-8674d957bc-bhcjr\" (UID: \"63f16300-0736-4975-ad35-2c6a5876338d\") " pod="openshift-console/console-8674d957bc-bhcjr" Jul 07 14:39:18.889250 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:18.889098 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/63f16300-0736-4975-ad35-2c6a5876338d-service-ca\") pod \"console-8674d957bc-bhcjr\" (UID: \"63f16300-0736-4975-ad35-2c6a5876338d\") " pod="openshift-console/console-8674d957bc-bhcjr" Jul 07 14:39:18.889250 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:18.889135 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/63f16300-0736-4975-ad35-2c6a5876338d-oauth-serving-cert\") pod \"console-8674d957bc-bhcjr\" (UID: \"63f16300-0736-4975-ad35-2c6a5876338d\") " pod="openshift-console/console-8674d957bc-bhcjr" Jul 07 14:39:18.889421 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:18.889360 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8tdk\" (UniqueName: \"kubernetes.io/projected/63f16300-0736-4975-ad35-2c6a5876338d-kube-api-access-d8tdk\") pod \"console-8674d957bc-bhcjr\" (UID: \"63f16300-0736-4975-ad35-2c6a5876338d\") " pod="openshift-console/console-8674d957bc-bhcjr" Jul 07 14:39:18.889421 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:18.889399 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63f16300-0736-4975-ad35-2c6a5876338d-trusted-ca-bundle\") pod \"console-8674d957bc-bhcjr\" (UID: \"63f16300-0736-4975-ad35-2c6a5876338d\") " pod="openshift-console/console-8674d957bc-bhcjr" Jul 07 14:39:18.889518 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:18.889433 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/63f16300-0736-4975-ad35-2c6a5876338d-console-oauth-config\") pod \"console-8674d957bc-bhcjr\" (UID: \"63f16300-0736-4975-ad35-2c6a5876338d\") " pod="openshift-console/console-8674d957bc-bhcjr" Jul 07 14:39:18.897391 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:18.897367 2555 kubelet.go:1648] "Image garbage collection succeeded" Jul 07 14:39:18.989814 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:18.989785 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8tdk\" (UniqueName: \"kubernetes.io/projected/63f16300-0736-4975-ad35-2c6a5876338d-kube-api-access-d8tdk\") pod \"console-8674d957bc-bhcjr\" (UID: \"63f16300-0736-4975-ad35-2c6a5876338d\") " pod="openshift-console/console-8674d957bc-bhcjr" Jul 07 14:39:18.990238 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:18.989821 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63f16300-0736-4975-ad35-2c6a5876338d-trusted-ca-bundle\") pod \"console-8674d957bc-bhcjr\" (UID: \"63f16300-0736-4975-ad35-2c6a5876338d\") " pod="openshift-console/console-8674d957bc-bhcjr" Jul 07 14:39:18.990238 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:18.989845 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/63f16300-0736-4975-ad35-2c6a5876338d-console-oauth-config\") pod \"console-8674d957bc-bhcjr\" (UID: \"63f16300-0736-4975-ad35-2c6a5876338d\") " pod="openshift-console/console-8674d957bc-bhcjr" Jul 07 14:39:18.990238 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:18.989887 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/63f16300-0736-4975-ad35-2c6a5876338d-console-serving-cert\") pod \"console-8674d957bc-bhcjr\" (UID: \"63f16300-0736-4975-ad35-2c6a5876338d\") " pod="openshift-console/console-8674d957bc-bhcjr" Jul 07 14:39:18.990238 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:18.989933 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/63f16300-0736-4975-ad35-2c6a5876338d-console-config\") pod \"console-8674d957bc-bhcjr\" (UID: \"63f16300-0736-4975-ad35-2c6a5876338d\") " pod="openshift-console/console-8674d957bc-bhcjr" Jul 07 14:39:18.990238 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:18.989963 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/63f16300-0736-4975-ad35-2c6a5876338d-service-ca\") pod \"console-8674d957bc-bhcjr\" (UID: \"63f16300-0736-4975-ad35-2c6a5876338d\") " pod="openshift-console/console-8674d957bc-bhcjr" Jul 07 14:39:18.990238 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:18.989986 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/63f16300-0736-4975-ad35-2c6a5876338d-oauth-serving-cert\") pod \"console-8674d957bc-bhcjr\" (UID: \"63f16300-0736-4975-ad35-2c6a5876338d\") " pod="openshift-console/console-8674d957bc-bhcjr" Jul 07 14:39:18.992603 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:18.992581 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 07 14:39:18.992817 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:18.992801 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 07 14:39:18.992897 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:18.992878 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 07 14:39:18.992958 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:18.992936 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 07 14:39:18.993065 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:18.993048 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 07 14:39:18.998107 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:18.998092 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 07 14:39:18.998861 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:18.998825 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 07 14:39:19.001156 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:19.001130 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/63f16300-0736-4975-ad35-2c6a5876338d-console-config\") pod \"console-8674d957bc-bhcjr\" (UID: \"63f16300-0736-4975-ad35-2c6a5876338d\") " pod="openshift-console/console-8674d957bc-bhcjr" Jul 07 14:39:19.001251 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:19.001214 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/63f16300-0736-4975-ad35-2c6a5876338d-service-ca\") pod \"console-8674d957bc-bhcjr\" (UID: \"63f16300-0736-4975-ad35-2c6a5876338d\") " pod="openshift-console/console-8674d957bc-bhcjr" Jul 07 14:39:19.001316 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:19.001301 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/63f16300-0736-4975-ad35-2c6a5876338d-oauth-serving-cert\") pod \"console-8674d957bc-bhcjr\" (UID: \"63f16300-0736-4975-ad35-2c6a5876338d\") " pod="openshift-console/console-8674d957bc-bhcjr" Jul 07 14:39:19.001407 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:19.001391 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63f16300-0736-4975-ad35-2c6a5876338d-trusted-ca-bundle\") pod \"console-8674d957bc-bhcjr\" (UID: \"63f16300-0736-4975-ad35-2c6a5876338d\") " pod="openshift-console/console-8674d957bc-bhcjr" Jul 07 14:39:19.002843 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:19.002817 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/63f16300-0736-4975-ad35-2c6a5876338d-console-oauth-config\") pod \"console-8674d957bc-bhcjr\" (UID: \"63f16300-0736-4975-ad35-2c6a5876338d\") " pod="openshift-console/console-8674d957bc-bhcjr" Jul 07 14:39:19.002947 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:19.002932 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/63f16300-0736-4975-ad35-2c6a5876338d-console-serving-cert\") pod \"console-8674d957bc-bhcjr\" (UID: \"63f16300-0736-4975-ad35-2c6a5876338d\") " pod="openshift-console/console-8674d957bc-bhcjr" Jul 07 14:39:19.009945 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:19.009928 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 07 14:39:19.020231 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:19.020208 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8tdk\" (UniqueName: \"kubernetes.io/projected/63f16300-0736-4975-ad35-2c6a5876338d-kube-api-access-d8tdk\") pod \"console-8674d957bc-bhcjr\" (UID: \"63f16300-0736-4975-ad35-2c6a5876338d\") " pod="openshift-console/console-8674d957bc-bhcjr" Jul 07 14:39:19.062339 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:19.062210 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7svhg\"" Jul 07 14:39:19.074435 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:19.070766 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8674d957bc-bhcjr" Jul 07 14:39:19.196354 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:19.196331 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8674d957bc-bhcjr"] Jul 07 14:39:19.198669 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:39:19.198644 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63f16300_0736_4975_ad35_2c6a5876338d.slice/crio-4d135580218f5f03c482d4736fa15a1b697419373288125beca7af0c91535514 WatchSource:0}: Error finding container 4d135580218f5f03c482d4736fa15a1b697419373288125beca7af0c91535514: Status 404 returned error can't find the container with id 4d135580218f5f03c482d4736fa15a1b697419373288125beca7af0c91535514 Jul 07 14:39:19.993998 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:19.993964 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8674d957bc-bhcjr" event={"ID":"63f16300-0736-4975-ad35-2c6a5876338d","Type":"ContainerStarted","Data":"4d135580218f5f03c482d4736fa15a1b697419373288125beca7af0c91535514"} Jul 07 14:39:23.011364 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:23.011320 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8674d957bc-bhcjr" event={"ID":"63f16300-0736-4975-ad35-2c6a5876338d","Type":"ContainerStarted","Data":"667fd77629f7eba05c08c2e838eb13890ce26392ca4443080b6b48766d1fda2e"} Jul 07 14:39:23.031867 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:23.031820 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8674d957bc-bhcjr" podStartSLOduration=2.187855285 podStartE2EDuration="5.031805407s" totalImagesPullingTime="2.843950122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:39:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:39:23.030413319 +0000 UTC m=+304.454599794" watchObservedRunningTime="2026-07-07 14:39:23.031805407 +0000 UTC m=+304.455991878" Jul 07 14:39:23.682032 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:23.682003 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6d8677489b-52crb" Jul 07 14:39:26.036801 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:26.036736 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpmtl" Jul 07 14:39:27.960711 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:27.960681 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jbflt"] Jul 07 14:39:27.966335 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:27.966313 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jbflt" Jul 07 14:39:27.969484 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:27.969455 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 07 14:39:27.970595 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:27.970570 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 07 14:39:27.970722 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:27.970612 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-zz49b\"" Jul 07 14:39:27.970722 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:27.970615 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 07 14:39:27.970722 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:27.970673 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 07 14:39:27.975801 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:27.975779 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xhlln"] Jul 07 14:39:27.979188 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:27.979169 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jbflt"] Jul 07 14:39:27.979299 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:27.979284 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xhlln" Jul 07 14:39:27.982214 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:27.982196 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 07 14:39:27.982408 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:27.982389 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 07 14:39:27.982457 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:27.982414 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 07 14:39:27.982510 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:27.982490 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2kmmz\"" Jul 07 14:39:28.067264 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.067237 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/afa77f75-9627-4138-9d5c-83e060022fdf-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-jbflt\" (UID: \"afa77f75-9627-4138-9d5c-83e060022fdf\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jbflt" Jul 07 14:39:28.067442 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.067270 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/afa77f75-9627-4138-9d5c-83e060022fdf-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-jbflt\" (UID: \"afa77f75-9627-4138-9d5c-83e060022fdf\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jbflt" Jul 07 14:39:28.067442 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.067289 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6b68f320-00d3-41cb-a818-39d5f50c6d07-root\") pod \"node-exporter-xhlln\" (UID: \"6b68f320-00d3-41cb-a818-39d5f50c6d07\") " pod="openshift-monitoring/node-exporter-xhlln" Jul 07 14:39:28.067442 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.067313 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/afa77f75-9627-4138-9d5c-83e060022fdf-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-jbflt\" (UID: \"afa77f75-9627-4138-9d5c-83e060022fdf\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jbflt" Jul 07 14:39:28.067442 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.067340 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcqjs\" (UniqueName: \"kubernetes.io/projected/afa77f75-9627-4138-9d5c-83e060022fdf-kube-api-access-lcqjs\") pod \"kube-state-metrics-7c9dbb8b7d-jbflt\" (UID: \"afa77f75-9627-4138-9d5c-83e060022fdf\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jbflt" Jul 07 14:39:28.067442 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.067376 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6b68f320-00d3-41cb-a818-39d5f50c6d07-node-exporter-wtmp\") pod \"node-exporter-xhlln\" (UID: \"6b68f320-00d3-41cb-a818-39d5f50c6d07\") " pod="openshift-monitoring/node-exporter-xhlln" Jul 07 14:39:28.067442 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.067430 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b68f320-00d3-41cb-a818-39d5f50c6d07-sys\") pod \"node-exporter-xhlln\" (UID: \"6b68f320-00d3-41cb-a818-39d5f50c6d07\") " pod="openshift-monitoring/node-exporter-xhlln" Jul 07 14:39:28.067682 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.067458 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lcck\" (UniqueName: \"kubernetes.io/projected/6b68f320-00d3-41cb-a818-39d5f50c6d07-kube-api-access-9lcck\") pod \"node-exporter-xhlln\" (UID: \"6b68f320-00d3-41cb-a818-39d5f50c6d07\") " pod="openshift-monitoring/node-exporter-xhlln" Jul 07 14:39:28.067682 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.067510 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b68f320-00d3-41cb-a818-39d5f50c6d07-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xhlln\" (UID: \"6b68f320-00d3-41cb-a818-39d5f50c6d07\") " pod="openshift-monitoring/node-exporter-xhlln" Jul 07 14:39:28.067682 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.067537 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/afa77f75-9627-4138-9d5c-83e060022fdf-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-jbflt\" (UID: \"afa77f75-9627-4138-9d5c-83e060022fdf\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jbflt" Jul 07 14:39:28.067682 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.067560 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b68f320-00d3-41cb-a818-39d5f50c6d07-metrics-client-ca\") pod \"node-exporter-xhlln\" (UID: \"6b68f320-00d3-41cb-a818-39d5f50c6d07\") " pod="openshift-monitoring/node-exporter-xhlln" Jul 07 14:39:28.067682 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.067579 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/afa77f75-9627-4138-9d5c-83e060022fdf-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-jbflt\" (UID: \"afa77f75-9627-4138-9d5c-83e060022fdf\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jbflt" Jul 07 14:39:28.067682 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.067600 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6b68f320-00d3-41cb-a818-39d5f50c6d07-node-exporter-tls\") pod \"node-exporter-xhlln\" (UID: \"6b68f320-00d3-41cb-a818-39d5f50c6d07\") " pod="openshift-monitoring/node-exporter-xhlln" Jul 07 14:39:28.067682 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.067621 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6b68f320-00d3-41cb-a818-39d5f50c6d07-node-exporter-textfile\") pod \"node-exporter-xhlln\" (UID: \"6b68f320-00d3-41cb-a818-39d5f50c6d07\") " pod="openshift-monitoring/node-exporter-xhlln" Jul 07 14:39:28.067682 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.067636 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6b68f320-00d3-41cb-a818-39d5f50c6d07-node-exporter-accelerators-collector-config\") pod \"node-exporter-xhlln\" (UID: \"6b68f320-00d3-41cb-a818-39d5f50c6d07\") " pod="openshift-monitoring/node-exporter-xhlln" Jul 07 14:39:28.169048 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.169014 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/afa77f75-9627-4138-9d5c-83e060022fdf-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-jbflt\" (UID: \"afa77f75-9627-4138-9d5c-83e060022fdf\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jbflt" Jul 07 14:39:28.169048 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.169048 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b68f320-00d3-41cb-a818-39d5f50c6d07-metrics-client-ca\") pod \"node-exporter-xhlln\" (UID: \"6b68f320-00d3-41cb-a818-39d5f50c6d07\") " pod="openshift-monitoring/node-exporter-xhlln" Jul 07 14:39:28.169279 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.169069 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/afa77f75-9627-4138-9d5c-83e060022fdf-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-jbflt\" (UID: \"afa77f75-9627-4138-9d5c-83e060022fdf\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jbflt" Jul 07 14:39:28.169279 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.169101 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6b68f320-00d3-41cb-a818-39d5f50c6d07-node-exporter-tls\") pod \"node-exporter-xhlln\" (UID: \"6b68f320-00d3-41cb-a818-39d5f50c6d07\") " pod="openshift-monitoring/node-exporter-xhlln" Jul 07 14:39:28.169279 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.169140 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6b68f320-00d3-41cb-a818-39d5f50c6d07-node-exporter-textfile\") pod \"node-exporter-xhlln\" (UID: \"6b68f320-00d3-41cb-a818-39d5f50c6d07\") " pod="openshift-monitoring/node-exporter-xhlln" Jul 07 14:39:28.169279 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.169164 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6b68f320-00d3-41cb-a818-39d5f50c6d07-node-exporter-accelerators-collector-config\") pod \"node-exporter-xhlln\" (UID: \"6b68f320-00d3-41cb-a818-39d5f50c6d07\") " pod="openshift-monitoring/node-exporter-xhlln" Jul 07 14:39:28.169279 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:39:28.169203 2555 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 07 14:39:28.169536 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:39:28.169287 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afa77f75-9627-4138-9d5c-83e060022fdf-kube-state-metrics-tls podName:afa77f75-9627-4138-9d5c-83e060022fdf nodeName:}" failed. No retries permitted until 2026-07-07 14:39:28.669266487 +0000 UTC m=+310.093452960 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/afa77f75-9627-4138-9d5c-83e060022fdf-kube-state-metrics-tls") pod "kube-state-metrics-7c9dbb8b7d-jbflt" (UID: "afa77f75-9627-4138-9d5c-83e060022fdf") : secret "kube-state-metrics-tls" not found Jul 07 14:39:28.169536 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.169496 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6b68f320-00d3-41cb-a818-39d5f50c6d07-node-exporter-textfile\") pod \"node-exporter-xhlln\" (UID: \"6b68f320-00d3-41cb-a818-39d5f50c6d07\") " pod="openshift-monitoring/node-exporter-xhlln" Jul 07 14:39:28.169536 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.169206 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/afa77f75-9627-4138-9d5c-83e060022fdf-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-jbflt\" (UID: \"afa77f75-9627-4138-9d5c-83e060022fdf\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jbflt" Jul 07 14:39:28.169536 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.169524 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/afa77f75-9627-4138-9d5c-83e060022fdf-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-jbflt\" (UID: \"afa77f75-9627-4138-9d5c-83e060022fdf\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jbflt" Jul 07 14:39:28.169787 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.169560 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/afa77f75-9627-4138-9d5c-83e060022fdf-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-jbflt\" (UID: \"afa77f75-9627-4138-9d5c-83e060022fdf\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jbflt" Jul 07 14:39:28.169787 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.169599 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6b68f320-00d3-41cb-a818-39d5f50c6d07-root\") pod \"node-exporter-xhlln\" (UID: \"6b68f320-00d3-41cb-a818-39d5f50c6d07\") " pod="openshift-monitoring/node-exporter-xhlln" Jul 07 14:39:28.169787 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.169640 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/afa77f75-9627-4138-9d5c-83e060022fdf-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-jbflt\" (UID: \"afa77f75-9627-4138-9d5c-83e060022fdf\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jbflt" Jul 07 14:39:28.169787 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.169690 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcqjs\" (UniqueName: \"kubernetes.io/projected/afa77f75-9627-4138-9d5c-83e060022fdf-kube-api-access-lcqjs\") pod \"kube-state-metrics-7c9dbb8b7d-jbflt\" (UID: \"afa77f75-9627-4138-9d5c-83e060022fdf\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jbflt" Jul 07 14:39:28.169787 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.169715 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6b68f320-00d3-41cb-a818-39d5f50c6d07-node-exporter-wtmp\") pod \"node-exporter-xhlln\" (UID: \"6b68f320-00d3-41cb-a818-39d5f50c6d07\") " pod="openshift-monitoring/node-exporter-xhlln" Jul 07 14:39:28.169787 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.169726 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6b68f320-00d3-41cb-a818-39d5f50c6d07-root\") pod \"node-exporter-xhlln\" (UID: \"6b68f320-00d3-41cb-a818-39d5f50c6d07\") " pod="openshift-monitoring/node-exporter-xhlln" Jul 07 14:39:28.169787 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.169784 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b68f320-00d3-41cb-a818-39d5f50c6d07-sys\") pod \"node-exporter-xhlln\" (UID: \"6b68f320-00d3-41cb-a818-39d5f50c6d07\") " pod="openshift-monitoring/node-exporter-xhlln" Jul 07 14:39:28.170158 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.169813 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lcck\" (UniqueName: \"kubernetes.io/projected/6b68f320-00d3-41cb-a818-39d5f50c6d07-kube-api-access-9lcck\") pod \"node-exporter-xhlln\" (UID: \"6b68f320-00d3-41cb-a818-39d5f50c6d07\") " pod="openshift-monitoring/node-exporter-xhlln" Jul 07 14:39:28.170158 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.169848 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b68f320-00d3-41cb-a818-39d5f50c6d07-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xhlln\" (UID: \"6b68f320-00d3-41cb-a818-39d5f50c6d07\") " pod="openshift-monitoring/node-exporter-xhlln" Jul 07 14:39:28.170158 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.170037 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6b68f320-00d3-41cb-a818-39d5f50c6d07-node-exporter-wtmp\") pod \"node-exporter-xhlln\" (UID: \"6b68f320-00d3-41cb-a818-39d5f50c6d07\") " pod="openshift-monitoring/node-exporter-xhlln" Jul 07 14:39:28.170158 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.170082 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b68f320-00d3-41cb-a818-39d5f50c6d07-sys\") pod \"node-exporter-xhlln\" (UID: \"6b68f320-00d3-41cb-a818-39d5f50c6d07\") " pod="openshift-monitoring/node-exporter-xhlln" Jul 07 14:39:28.170158 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.170146 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6b68f320-00d3-41cb-a818-39d5f50c6d07-node-exporter-accelerators-collector-config\") pod \"node-exporter-xhlln\" (UID: \"6b68f320-00d3-41cb-a818-39d5f50c6d07\") " pod="openshift-monitoring/node-exporter-xhlln" Jul 07 14:39:28.170479 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.170446 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/afa77f75-9627-4138-9d5c-83e060022fdf-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-jbflt\" (UID: \"afa77f75-9627-4138-9d5c-83e060022fdf\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jbflt" Jul 07 14:39:28.170664 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.170513 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/afa77f75-9627-4138-9d5c-83e060022fdf-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-jbflt\" (UID: \"afa77f75-9627-4138-9d5c-83e060022fdf\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jbflt" Jul 07 14:39:28.170819 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.170707 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b68f320-00d3-41cb-a818-39d5f50c6d07-metrics-client-ca\") pod \"node-exporter-xhlln\" (UID: \"6b68f320-00d3-41cb-a818-39d5f50c6d07\") " pod="openshift-monitoring/node-exporter-xhlln" Jul 07 14:39:28.172217 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.172192 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/afa77f75-9627-4138-9d5c-83e060022fdf-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-jbflt\" (UID: \"afa77f75-9627-4138-9d5c-83e060022fdf\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jbflt" Jul 07 14:39:28.172776 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.172720 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6b68f320-00d3-41cb-a818-39d5f50c6d07-node-exporter-tls\") pod \"node-exporter-xhlln\" (UID: \"6b68f320-00d3-41cb-a818-39d5f50c6d07\") " pod="openshift-monitoring/node-exporter-xhlln" Jul 07 14:39:28.172953 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.172935 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b68f320-00d3-41cb-a818-39d5f50c6d07-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xhlln\" (UID: \"6b68f320-00d3-41cb-a818-39d5f50c6d07\") " pod="openshift-monitoring/node-exporter-xhlln" Jul 07 14:39:28.178269 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.178251 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lcck\" (UniqueName: \"kubernetes.io/projected/6b68f320-00d3-41cb-a818-39d5f50c6d07-kube-api-access-9lcck\") pod \"node-exporter-xhlln\" (UID: \"6b68f320-00d3-41cb-a818-39d5f50c6d07\") " pod="openshift-monitoring/node-exporter-xhlln" Jul 07 14:39:28.179002 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.178984 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcqjs\" (UniqueName: \"kubernetes.io/projected/afa77f75-9627-4138-9d5c-83e060022fdf-kube-api-access-lcqjs\") pod \"kube-state-metrics-7c9dbb8b7d-jbflt\" (UID: \"afa77f75-9627-4138-9d5c-83e060022fdf\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jbflt" Jul 07 14:39:28.288171 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.288092 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xhlln" Jul 07 14:39:28.300348 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:39:28.300324 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b68f320_00d3_41cb_a818_39d5f50c6d07.slice/crio-5af58983d0f20c2a9e325bb7c276fa487571264677ddae470fdf7d4af56f0a29 WatchSource:0}: Error finding container 5af58983d0f20c2a9e325bb7c276fa487571264677ddae470fdf7d4af56f0a29: Status 404 returned error can't find the container with id 5af58983d0f20c2a9e325bb7c276fa487571264677ddae470fdf7d4af56f0a29 Jul 07 14:39:28.302073 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.302051 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 14:39:28.673907 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.673875 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/afa77f75-9627-4138-9d5c-83e060022fdf-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-jbflt\" (UID: \"afa77f75-9627-4138-9d5c-83e060022fdf\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jbflt" Jul 07 14:39:28.676352 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.676323 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/afa77f75-9627-4138-9d5c-83e060022fdf-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-jbflt\" (UID: \"afa77f75-9627-4138-9d5c-83e060022fdf\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jbflt" Jul 07 14:39:28.697479 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.697438 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6d8677489b-52crb" podUID="18ec5f4a-e826-4534-a068-3fb383136ca1" containerName="registry" containerID="cri-o://0d8cba06bdc6cc565705081fae435e15c4ae75bfda5ef1b73349c3007edb338c" gracePeriod=30 Jul 07 14:39:28.875737 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.875284 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jbflt" Jul 07 14:39:28.953363 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:28.953286 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d8677489b-52crb" Jul 07 14:39:29.015465 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.015015 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 14:39:29.015465 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.015367 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18ec5f4a-e826-4534-a068-3fb383136ca1" containerName="registry" Jul 07 14:39:29.015465 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.015380 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="18ec5f4a-e826-4534-a068-3fb383136ca1" containerName="registry" Jul 07 14:39:29.015465 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.015445 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="18ec5f4a-e826-4534-a068-3fb383136ca1" containerName="registry" Jul 07 14:39:29.019100 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.019079 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:39:29.022036 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.021770 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 07 14:39:29.022036 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.021906 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 07 14:39:29.022036 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.021920 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 07 14:39:29.022036 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.022037 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 07 14:39:29.022342 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.022153 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ssm4h\"" Jul 07 14:39:29.022342 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.022199 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 07 14:39:29.022342 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.022319 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 07 14:39:29.022512 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.022433 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 07 14:39:29.022512 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.022456 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 07 14:39:29.022812 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.022742 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 07 14:39:29.033679 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.033550 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xhlln" event={"ID":"6b68f320-00d3-41cb-a818-39d5f50c6d07","Type":"ContainerStarted","Data":"5af58983d0f20c2a9e325bb7c276fa487571264677ddae470fdf7d4af56f0a29"} Jul 07 14:39:29.034847 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.034825 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 14:39:29.035184 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.035161 2555 generic.go:356] "Generic (PLEG): container finished" podID="18ec5f4a-e826-4534-a068-3fb383136ca1" containerID="0d8cba06bdc6cc565705081fae435e15c4ae75bfda5ef1b73349c3007edb338c" exitCode=0 Jul 07 14:39:29.035290 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.035205 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d8677489b-52crb" Jul 07 14:39:29.035290 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.035218 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d8677489b-52crb" event={"ID":"18ec5f4a-e826-4534-a068-3fb383136ca1","Type":"ContainerDied","Data":"0d8cba06bdc6cc565705081fae435e15c4ae75bfda5ef1b73349c3007edb338c"} Jul 07 14:39:29.035290 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.035262 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d8677489b-52crb" event={"ID":"18ec5f4a-e826-4534-a068-3fb383136ca1","Type":"ContainerDied","Data":"f8ec536cc95426a1dabbef2833b52842d265d90c442b5fccfbcbe13c4897a036"} Jul 07 14:39:29.035290 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.035288 2555 scope.go:117] "RemoveContainer" containerID="0d8cba06bdc6cc565705081fae435e15c4ae75bfda5ef1b73349c3007edb338c" Jul 07 14:39:29.044204 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.044186 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jbflt"] Jul 07 14:39:29.044331 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:39:29.044305 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafa77f75_9627_4138_9d5c_83e060022fdf.slice/crio-fd5352db2c35037af7bf3cd6645657775649aff72419c5572a7c3bd58a2d7477 WatchSource:0}: Error finding container fd5352db2c35037af7bf3cd6645657775649aff72419c5572a7c3bd58a2d7477: Status 404 returned error can't find the container with id fd5352db2c35037af7bf3cd6645657775649aff72419c5572a7c3bd58a2d7477 Jul 07 14:39:29.047639 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.047623 2555 scope.go:117] "RemoveContainer" containerID="0d8cba06bdc6cc565705081fae435e15c4ae75bfda5ef1b73349c3007edb338c" Jul 07 14:39:29.048053 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:39:29.048030 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d8cba06bdc6cc565705081fae435e15c4ae75bfda5ef1b73349c3007edb338c\": container with ID starting with 0d8cba06bdc6cc565705081fae435e15c4ae75bfda5ef1b73349c3007edb338c not found: ID does not exist" containerID="0d8cba06bdc6cc565705081fae435e15c4ae75bfda5ef1b73349c3007edb338c" Jul 07 14:39:29.048265 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.048214 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d8cba06bdc6cc565705081fae435e15c4ae75bfda5ef1b73349c3007edb338c"} err="failed to get container status \"0d8cba06bdc6cc565705081fae435e15c4ae75bfda5ef1b73349c3007edb338c\": rpc error: code = NotFound desc = could not find container \"0d8cba06bdc6cc565705081fae435e15c4ae75bfda5ef1b73349c3007edb338c\": container with ID starting with 0d8cba06bdc6cc565705081fae435e15c4ae75bfda5ef1b73349c3007edb338c not found: ID does not exist" Jul 07 14:39:29.071390 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.071368 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8674d957bc-bhcjr" Jul 07 14:39:29.071507 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.071412 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8674d957bc-bhcjr" Jul 07 14:39:29.077307 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.077282 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/18ec5f4a-e826-4534-a068-3fb383136ca1-trusted-ca\") pod \"18ec5f4a-e826-4534-a068-3fb383136ca1\" (UID: \"18ec5f4a-e826-4534-a068-3fb383136ca1\") " Jul 07 14:39:29.077411 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.077328 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdtp2\" (UniqueName: \"kubernetes.io/projected/18ec5f4a-e826-4534-a068-3fb383136ca1-kube-api-access-mdtp2\") pod \"18ec5f4a-e826-4534-a068-3fb383136ca1\" (UID: \"18ec5f4a-e826-4534-a068-3fb383136ca1\") " Jul 07 14:39:29.077411 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.077381 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/18ec5f4a-e826-4534-a068-3fb383136ca1-bound-sa-token\") pod \"18ec5f4a-e826-4534-a068-3fb383136ca1\" (UID: \"18ec5f4a-e826-4534-a068-3fb383136ca1\") " Jul 07 14:39:29.077518 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.077408 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18ec5f4a-e826-4534-a068-3fb383136ca1-registry-tls\") pod \"18ec5f4a-e826-4534-a068-3fb383136ca1\" (UID: \"18ec5f4a-e826-4534-a068-3fb383136ca1\") " Jul 07 14:39:29.077518 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.077491 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/18ec5f4a-e826-4534-a068-3fb383136ca1-ca-trust-extracted\") pod \"18ec5f4a-e826-4534-a068-3fb383136ca1\" (UID: \"18ec5f4a-e826-4534-a068-3fb383136ca1\") " Jul 07 14:39:29.077612 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.077537 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/18ec5f4a-e826-4534-a068-3fb383136ca1-image-registry-private-configuration\") pod \"18ec5f4a-e826-4534-a068-3fb383136ca1\" (UID: \"18ec5f4a-e826-4534-a068-3fb383136ca1\") " Jul 07 14:39:29.077612 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.077574 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/18ec5f4a-e826-4534-a068-3fb383136ca1-installation-pull-secrets\") pod \"18ec5f4a-e826-4534-a068-3fb383136ca1\" (UID: \"18ec5f4a-e826-4534-a068-3fb383136ca1\") " Jul 07 14:39:29.077711 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.077641 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/18ec5f4a-e826-4534-a068-3fb383136ca1-registry-certificates\") pod \"18ec5f4a-e826-4534-a068-3fb383136ca1\" (UID: \"18ec5f4a-e826-4534-a068-3fb383136ca1\") " Jul 07 14:39:29.077874 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.077844 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18ec5f4a-e826-4534-a068-3fb383136ca1-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "18ec5f4a-e826-4534-a068-3fb383136ca1" (UID: "18ec5f4a-e826-4534-a068-3fb383136ca1"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:39:29.078009 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.077990 2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/18ec5f4a-e826-4534-a068-3fb383136ca1-trusted-ca\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:39:29.078129 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.078112 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8674d957bc-bhcjr" Jul 07 14:39:29.078436 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.078411 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18ec5f4a-e826-4534-a068-3fb383136ca1-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "18ec5f4a-e826-4534-a068-3fb383136ca1" (UID: "18ec5f4a-e826-4534-a068-3fb383136ca1"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:39:29.080280 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.080255 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18ec5f4a-e826-4534-a068-3fb383136ca1-kube-api-access-mdtp2" (OuterVolumeSpecName: "kube-api-access-mdtp2") pod "18ec5f4a-e826-4534-a068-3fb383136ca1" (UID: "18ec5f4a-e826-4534-a068-3fb383136ca1"). InnerVolumeSpecName "kube-api-access-mdtp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:39:29.081204 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.081167 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18ec5f4a-e826-4534-a068-3fb383136ca1-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "18ec5f4a-e826-4534-a068-3fb383136ca1" (UID: "18ec5f4a-e826-4534-a068-3fb383136ca1"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:39:29.081294 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.081256 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18ec5f4a-e826-4534-a068-3fb383136ca1-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "18ec5f4a-e826-4534-a068-3fb383136ca1" (UID: "18ec5f4a-e826-4534-a068-3fb383136ca1"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:39:29.081631 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.081609 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18ec5f4a-e826-4534-a068-3fb383136ca1-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "18ec5f4a-e826-4534-a068-3fb383136ca1" (UID: "18ec5f4a-e826-4534-a068-3fb383136ca1"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:39:29.081721 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.081695 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18ec5f4a-e826-4534-a068-3fb383136ca1-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "18ec5f4a-e826-4534-a068-3fb383136ca1" (UID: "18ec5f4a-e826-4534-a068-3fb383136ca1"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:39:29.089012 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.088989 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18ec5f4a-e826-4534-a068-3fb383136ca1-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "18ec5f4a-e826-4534-a068-3fb383136ca1" (UID: "18ec5f4a-e826-4534-a068-3fb383136ca1"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:39:29.179339 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.179286 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/42703fd0-9634-450d-b3a3-0dd97006d417-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:39:29.179460 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.179334 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/42703fd0-9634-450d-b3a3-0dd97006d417-config-volume\") pod \"alertmanager-main-0\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:39:29.179460 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.179408 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/42703fd0-9634-450d-b3a3-0dd97006d417-tls-assets\") pod \"alertmanager-main-0\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:39:29.179690 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.179471 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/42703fd0-9634-450d-b3a3-0dd97006d417-web-config\") pod \"alertmanager-main-0\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:39:29.179690 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.179514 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/42703fd0-9634-450d-b3a3-0dd97006d417-config-out\") pod \"alertmanager-main-0\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:39:29.179690 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.179627 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/42703fd0-9634-450d-b3a3-0dd97006d417-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:39:29.179960 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.179687 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/42703fd0-9634-450d-b3a3-0dd97006d417-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:39:29.179960 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.179727 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/42703fd0-9634-450d-b3a3-0dd97006d417-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:39:29.179960 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.179799 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/42703fd0-9634-450d-b3a3-0dd97006d417-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:39:29.179960 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.179826 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98lb8\" (UniqueName: \"kubernetes.io/projected/42703fd0-9634-450d-b3a3-0dd97006d417-kube-api-access-98lb8\") pod \"alertmanager-main-0\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:39:29.179960 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.179865 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42703fd0-9634-450d-b3a3-0dd97006d417-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:39:29.180329 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.180296 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42703fd0-9634-450d-b3a3-0dd97006d417-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:39:29.180390 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.180350 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/42703fd0-9634-450d-b3a3-0dd97006d417-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:39:29.180442 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.180395 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdtp2\" (UniqueName: \"kubernetes.io/projected/18ec5f4a-e826-4534-a068-3fb383136ca1-kube-api-access-mdtp2\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:39:29.180442 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.180413 2555 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/18ec5f4a-e826-4534-a068-3fb383136ca1-bound-sa-token\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:39:29.180543 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.180498 2555 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18ec5f4a-e826-4534-a068-3fb383136ca1-registry-tls\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:39:29.180543 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.180532 2555 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/18ec5f4a-e826-4534-a068-3fb383136ca1-ca-trust-extracted\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:39:29.180642 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.180550 2555 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/18ec5f4a-e826-4534-a068-3fb383136ca1-image-registry-private-configuration\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:39:29.180642 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.180590 2555 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/18ec5f4a-e826-4534-a068-3fb383136ca1-installation-pull-secrets\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:39:29.180642 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.180606 2555 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/18ec5f4a-e826-4534-a068-3fb383136ca1-registry-certificates\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:39:29.281353 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.281283 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/42703fd0-9634-450d-b3a3-0dd97006d417-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:39:29.281353 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.281316 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/42703fd0-9634-450d-b3a3-0dd97006d417-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:39:29.281353 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.281336 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/42703fd0-9634-450d-b3a3-0dd97006d417-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:39:29.281607 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.281381 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/42703fd0-9634-450d-b3a3-0dd97006d417-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:39:29.281607 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.281408 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98lb8\" (UniqueName: \"kubernetes.io/projected/42703fd0-9634-450d-b3a3-0dd97006d417-kube-api-access-98lb8\") pod \"alertmanager-main-0\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:39:29.281607 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.281444 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42703fd0-9634-450d-b3a3-0dd97006d417-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:39:29.281607 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:39:29.281468 2555 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 07 14:39:29.281607 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.281482 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42703fd0-9634-450d-b3a3-0dd97006d417-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:39:29.281607 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.281512 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/42703fd0-9634-450d-b3a3-0dd97006d417-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:39:29.281607 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:39:29.281532 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42703fd0-9634-450d-b3a3-0dd97006d417-secret-alertmanager-main-tls podName:42703fd0-9634-450d-b3a3-0dd97006d417 nodeName:}" failed. No retries permitted until 2026-07-07 14:39:29.781517264 +0000 UTC m=+311.205703717 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/42703fd0-9634-450d-b3a3-0dd97006d417-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "42703fd0-9634-450d-b3a3-0dd97006d417") : secret "alertmanager-main-tls" not found Jul 07 14:39:29.281607 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.281562 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/42703fd0-9634-450d-b3a3-0dd97006d417-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:39:29.281607 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.281583 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/42703fd0-9634-450d-b3a3-0dd97006d417-config-volume\") pod \"alertmanager-main-0\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:39:29.281607 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.281599 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/42703fd0-9634-450d-b3a3-0dd97006d417-tls-assets\") pod \"alertmanager-main-0\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:39:29.282109 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.281626 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/42703fd0-9634-450d-b3a3-0dd97006d417-web-config\") pod \"alertmanager-main-0\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:39:29.282109 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.281644 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/42703fd0-9634-450d-b3a3-0dd97006d417-config-out\") pod \"alertmanager-main-0\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:39:29.282109 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.281725 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/42703fd0-9634-450d-b3a3-0dd97006d417-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:39:29.282532 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.282502 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42703fd0-9634-450d-b3a3-0dd97006d417-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:39:29.283884 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.283230 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42703fd0-9634-450d-b3a3-0dd97006d417-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:39:29.284528 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.284500 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/42703fd0-9634-450d-b3a3-0dd97006d417-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:39:29.284620 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.284554 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/42703fd0-9634-450d-b3a3-0dd97006d417-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:39:29.284620 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.284570 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/42703fd0-9634-450d-b3a3-0dd97006d417-config-volume\") pod \"alertmanager-main-0\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:39:29.284802 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.284779 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/42703fd0-9634-450d-b3a3-0dd97006d417-config-out\") pod \"alertmanager-main-0\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:39:29.284963 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.284945 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/42703fd0-9634-450d-b3a3-0dd97006d417-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:39:29.285466 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.285449 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/42703fd0-9634-450d-b3a3-0dd97006d417-tls-assets\") pod \"alertmanager-main-0\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:39:29.286585 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.286566 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/42703fd0-9634-450d-b3a3-0dd97006d417-web-config\") pod \"alertmanager-main-0\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:39:29.286668 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.286624 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/42703fd0-9634-450d-b3a3-0dd97006d417-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:39:29.292156 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.292132 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98lb8\" (UniqueName: \"kubernetes.io/projected/42703fd0-9634-450d-b3a3-0dd97006d417-kube-api-access-98lb8\") pod \"alertmanager-main-0\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:39:29.366694 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.366669 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d8677489b-52crb"] Jul 07 14:39:29.375344 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.375324 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6d8677489b-52crb"] Jul 07 14:39:29.785716 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.785680 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/42703fd0-9634-450d-b3a3-0dd97006d417-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:39:29.788313 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.788286 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/42703fd0-9634-450d-b3a3-0dd97006d417-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:39:29.930582 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:29.930550 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:39:30.042151 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:30.042071 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jbflt" event={"ID":"afa77f75-9627-4138-9d5c-83e060022fdf","Type":"ContainerStarted","Data":"fd5352db2c35037af7bf3cd6645657775649aff72419c5572a7c3bd58a2d7477"} Jul 07 14:39:30.043916 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:30.043882 2555 generic.go:356] "Generic (PLEG): container finished" podID="6b68f320-00d3-41cb-a818-39d5f50c6d07" containerID="e7e03f28188d49f00384f127bae582c3459b9a0b007b58d7ab989f15f72f6cc5" exitCode=0 Jul 07 14:39:30.044043 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:30.043961 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xhlln" event={"ID":"6b68f320-00d3-41cb-a818-39d5f50c6d07","Type":"ContainerDied","Data":"e7e03f28188d49f00384f127bae582c3459b9a0b007b58d7ab989f15f72f6cc5"} Jul 07 14:39:30.049007 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:30.048987 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8674d957bc-bhcjr" Jul 07 14:39:30.076646 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:30.076626 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 14:39:30.275011 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:39:30.274970 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42703fd0_9634_450d_b3a3_0dd97006d417.slice/crio-84122df51d6db342f745ae9c382a9ebda7cd76e4b356fc77bf235803677e6c4e WatchSource:0}: Error finding container 84122df51d6db342f745ae9c382a9ebda7cd76e4b356fc77bf235803677e6c4e: Status 404 returned error can't find the container with id 84122df51d6db342f745ae9c382a9ebda7cd76e4b356fc77bf235803677e6c4e Jul 07 14:39:31.040875 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:31.040843 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18ec5f4a-e826-4534-a068-3fb383136ca1" path="/var/lib/kubelet/pods/18ec5f4a-e826-4534-a068-3fb383136ca1/volumes" Jul 07 14:39:31.049365 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:31.049332 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jbflt" event={"ID":"afa77f75-9627-4138-9d5c-83e060022fdf","Type":"ContainerStarted","Data":"27ef81412ccd39a79803b3dfca2f254739d0d346cb192c296517a527517d2dcb"} Jul 07 14:39:31.049738 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:31.049378 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jbflt" event={"ID":"afa77f75-9627-4138-9d5c-83e060022fdf","Type":"ContainerStarted","Data":"3c1a1d91d7d81a282482817a89b88eac171a954ca10b5d2f8c8065f32617b2b5"} Jul 07 14:39:31.049738 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:31.049398 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jbflt" event={"ID":"afa77f75-9627-4138-9d5c-83e060022fdf","Type":"ContainerStarted","Data":"d6be517f36aabc21e79392c19ac1ca14bedc7f6a240c524449bebf3845451db0"} Jul 07 14:39:31.050418 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:31.050392 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"42703fd0-9634-450d-b3a3-0dd97006d417","Type":"ContainerStarted","Data":"84122df51d6db342f745ae9c382a9ebda7cd76e4b356fc77bf235803677e6c4e"} Jul 07 14:39:31.052069 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:31.052049 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xhlln" event={"ID":"6b68f320-00d3-41cb-a818-39d5f50c6d07","Type":"ContainerStarted","Data":"61bf312441496a6419f39aee5b4986ba7454480be61a289fc049278935613217"} Jul 07 14:39:31.052159 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:31.052084 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xhlln" event={"ID":"6b68f320-00d3-41cb-a818-39d5f50c6d07","Type":"ContainerStarted","Data":"056282221c526d4b704356c49fa4bf49b02e98eb25e64f13e3c99787a9a13bf9"} Jul 07 14:39:31.083830 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:31.083783 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jbflt" podStartSLOduration=2.8102828239999997 podStartE2EDuration="4.083766408s" totalImagesPullingTime="1.273483584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:39:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:39:31.082417496 +0000 UTC m=+312.506603972" watchObservedRunningTime="2026-07-07 14:39:31.083766408 +0000 UTC m=+312.507952877" Jul 07 14:39:31.106688 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:31.106643 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xhlln" podStartSLOduration=2.974238399 podStartE2EDuration="4.106625542s" totalImagesPullingTime="1.069897123s" totalInitContainerRuntime="62.49002ms" isStatefulPod=true podCreationTimestamp="2026-07-07 14:39:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:39:31.104834626 +0000 UTC m=+312.529021101" watchObservedRunningTime="2026-07-07 14:39:31.106625542 +0000 UTC m=+312.530812014" Jul 07 14:39:32.057481 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:32.057447 2555 generic.go:356] "Generic (PLEG): container finished" podID="42703fd0-9634-450d-b3a3-0dd97006d417" containerID="8138979f89d45758b5e8103aa63800e5222211a13f37ac0674104a0ed221c57f" exitCode=0 Jul 07 14:39:32.057988 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:32.057539 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"42703fd0-9634-450d-b3a3-0dd97006d417","Type":"ContainerDied","Data":"8138979f89d45758b5e8103aa63800e5222211a13f37ac0674104a0ed221c57f"} Jul 07 14:39:34.068344 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.068315 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"42703fd0-9634-450d-b3a3-0dd97006d417","Type":"ContainerStarted","Data":"90c6641a7740bf6e5762b35cc3f6eed36f190e3316190d0ad8720cd292630682"} Jul 07 14:39:34.068704 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.068397 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"42703fd0-9634-450d-b3a3-0dd97006d417","Type":"ContainerStarted","Data":"b8744c20a7a9b812f69b4282f4a32e74f5fd0d88f2d23d8261db2ca1f82f05a1"} Jul 07 14:39:34.068704 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.068439 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"42703fd0-9634-450d-b3a3-0dd97006d417","Type":"ContainerStarted","Data":"0f22dbc31619d42ab74b8734e1ea2716a378b090c53be7670af12fe15c5cdefc"} Jul 07 14:39:34.068704 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.068476 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"42703fd0-9634-450d-b3a3-0dd97006d417","Type":"ContainerStarted","Data":"cbdf6bd18026e5ec68daae1139226f1f4fdea8f9ff58900d86457947eb091bb9"} Jul 07 14:39:34.068704 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.068509 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"42703fd0-9634-450d-b3a3-0dd97006d417","Type":"ContainerStarted","Data":"4a57cfef091150bd916a79aba28771bcbddbb31c27fd5a32eb7d5ac48b0fe5f5"} Jul 07 14:39:34.200802 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.200767 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 14:39:34.204536 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.204514 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.207062 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.207037 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 07 14:39:34.207559 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.207537 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 07 14:39:34.207738 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.207716 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 07 14:39:34.208169 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.208156 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 07 14:39:34.208823 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.208780 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 07 14:39:34.208823 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.208790 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 07 14:39:34.208990 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.208783 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 07 14:39:34.208990 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.208843 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 07 14:39:34.208990 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.208846 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 07 14:39:34.208990 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.208894 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 07 14:39:34.208990 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.208895 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 07 14:39:34.209230 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.209209 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 07 14:39:34.209286 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.209234 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-cdx4k\"" Jul 07 14:39:34.209286 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.209234 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-in0dqnal7t3k\"" Jul 07 14:39:34.209379 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.209213 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 07 14:39:34.209379 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.209234 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 07 14:39:34.211209 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.211187 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 07 14:39:34.223947 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.223925 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 14:39:34.331890 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.331868 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.332040 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.331903 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.332040 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.331924 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.332040 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.331954 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/45780cbd-0927-456b-9745-2cb2e5c8c347-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.332158 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.332031 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-web-config\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.332158 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.332064 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.332158 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.332085 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-config\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.332158 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.332113 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45780cbd-0927-456b-9745-2cb2e5c8c347-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.332158 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.332134 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45780cbd-0927-456b-9745-2cb2e5c8c347-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.332158 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.332154 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/45780cbd-0927-456b-9745-2cb2e5c8c347-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.332340 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.332171 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.332340 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.332222 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45780cbd-0927-456b-9745-2cb2e5c8c347-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.332340 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.332243 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/45780cbd-0927-456b-9745-2cb2e5c8c347-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.332340 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.332257 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlw7m\" (UniqueName: \"kubernetes.io/projected/45780cbd-0927-456b-9745-2cb2e5c8c347-kube-api-access-rlw7m\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.332340 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.332285 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.332340 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.332299 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/45780cbd-0927-456b-9745-2cb2e5c8c347-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.332340 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.332318 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/45780cbd-0927-456b-9745-2cb2e5c8c347-config-out\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.332340 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.332340 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45780cbd-0927-456b-9745-2cb2e5c8c347-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.332585 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.332358 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/45780cbd-0927-456b-9745-2cb2e5c8c347-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.332585 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.332377 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.433334 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.433303 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/45780cbd-0927-456b-9745-2cb2e5c8c347-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.433575 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.433339 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-web-config\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.433575 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.433360 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.433575 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.433379 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-config\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.433575 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.433405 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45780cbd-0927-456b-9745-2cb2e5c8c347-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.433575 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.433518 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45780cbd-0927-456b-9745-2cb2e5c8c347-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.433575 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.433560 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/45780cbd-0927-456b-9745-2cb2e5c8c347-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.434000 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.433583 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.434000 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.433629 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45780cbd-0927-456b-9745-2cb2e5c8c347-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.434000 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.433662 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/45780cbd-0927-456b-9745-2cb2e5c8c347-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.434000 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.433686 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlw7m\" (UniqueName: \"kubernetes.io/projected/45780cbd-0927-456b-9745-2cb2e5c8c347-kube-api-access-rlw7m\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.434000 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.433722 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.434000 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.433745 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/45780cbd-0927-456b-9745-2cb2e5c8c347-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.434294 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.434119 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/45780cbd-0927-456b-9745-2cb2e5c8c347-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.434294 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.434155 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/45780cbd-0927-456b-9745-2cb2e5c8c347-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.434294 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.434212 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45780cbd-0927-456b-9745-2cb2e5c8c347-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.434562 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.434537 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/45780cbd-0927-456b-9745-2cb2e5c8c347-config-out\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.434649 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.434583 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45780cbd-0927-456b-9745-2cb2e5c8c347-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.434649 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.434611 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/45780cbd-0927-456b-9745-2cb2e5c8c347-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.434792 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.434646 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.434792 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.434680 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.434792 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.434721 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.434953 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.434790 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.435132 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.434540 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45780cbd-0927-456b-9745-2cb2e5c8c347-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.435819 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.435792 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45780cbd-0927-456b-9745-2cb2e5c8c347-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.436014 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.435992 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/45780cbd-0927-456b-9745-2cb2e5c8c347-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.436372 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.434722 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45780cbd-0927-456b-9745-2cb2e5c8c347-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.437056 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.437021 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.437056 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.437035 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/45780cbd-0927-456b-9745-2cb2e5c8c347-config-out\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.437624 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.437600 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-config\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.437711 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.437658 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/45780cbd-0927-456b-9745-2cb2e5c8c347-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.438036 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.437984 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-web-config\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.438144 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.438097 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/45780cbd-0927-456b-9745-2cb2e5c8c347-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.438204 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.438189 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.438472 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.438448 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.438607 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.438591 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.439494 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.439468 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.439494 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.439476 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.439599 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.439537 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.442887 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.442867 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlw7m\" (UniqueName: \"kubernetes.io/projected/45780cbd-0927-456b-9745-2cb2e5c8c347-kube-api-access-rlw7m\") pod \"prometheus-k8s-0\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.515033 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.514992 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:34.653685 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:34.653655 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 14:39:34.656895 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:39:34.656866 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45780cbd_0927_456b_9745_2cb2e5c8c347.slice/crio-51e478b4b4866bb1d4158c70b06ae08e1cabbbcf69c80ff1f61be211fe8a57e2 WatchSource:0}: Error finding container 51e478b4b4866bb1d4158c70b06ae08e1cabbbcf69c80ff1f61be211fe8a57e2: Status 404 returned error can't find the container with id 51e478b4b4866bb1d4158c70b06ae08e1cabbbcf69c80ff1f61be211fe8a57e2 Jul 07 14:39:35.073929 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:35.073895 2555 generic.go:356] "Generic (PLEG): container finished" podID="45780cbd-0927-456b-9745-2cb2e5c8c347" containerID="33a445467b13a1f73e2c3f881972ae29dd6741352f979e8a0121df6ab068ee8c" exitCode=0 Jul 07 14:39:35.074333 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:35.073974 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45780cbd-0927-456b-9745-2cb2e5c8c347","Type":"ContainerDied","Data":"33a445467b13a1f73e2c3f881972ae29dd6741352f979e8a0121df6ab068ee8c"} Jul 07 14:39:35.074333 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:35.074061 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45780cbd-0927-456b-9745-2cb2e5c8c347","Type":"ContainerStarted","Data":"51e478b4b4866bb1d4158c70b06ae08e1cabbbcf69c80ff1f61be211fe8a57e2"} Jul 07 14:39:35.077562 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:35.077535 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"42703fd0-9634-450d-b3a3-0dd97006d417","Type":"ContainerStarted","Data":"7781de0af30ec08b7ddd50d16c8f1e55428628290cc8e823e867d1c804d561f2"} Jul 07 14:39:35.133141 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:35.133082 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.308296537 podStartE2EDuration="7.133065484s" totalImagesPullingTime="3.808958241s" totalInitContainerRuntime="15.810706ms" isStatefulPod=false podCreationTimestamp="2026-07-07 14:39:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:39:35.131163365 +0000 UTC m=+316.555349858" watchObservedRunningTime="2026-07-07 14:39:35.133065484 +0000 UTC m=+316.557251959" Jul 07 14:39:39.098000 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:39.097966 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45780cbd-0927-456b-9745-2cb2e5c8c347","Type":"ContainerStarted","Data":"5c972dee3f7ea1977c1ea3311e60b942f21feb2596824c17ac897f63ce96c078"} Jul 07 14:39:39.098363 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:39.098031 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45780cbd-0927-456b-9745-2cb2e5c8c347","Type":"ContainerStarted","Data":"8848d48786e07e17317074e50d6d368f2cd9e608751bb9e727d4ad14a49c10a9"} Jul 07 14:39:41.110893 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:41.110867 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45780cbd-0927-456b-9745-2cb2e5c8c347","Type":"ContainerStarted","Data":"d9831b9e57924d695de6e952962968795c1050a6c862f71e50f9fd13bbfbac9a"} Jul 07 14:39:41.111225 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:41.110929 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45780cbd-0927-456b-9745-2cb2e5c8c347","Type":"ContainerStarted","Data":"cf9cd0a46fbc4be92d5225e736bad570b59be29dff591f7e7e2ed032220d0ba7"} Jul 07 14:39:42.118717 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:42.118683 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45780cbd-0927-456b-9745-2cb2e5c8c347","Type":"ContainerStarted","Data":"983a0b90afd6d6be3d792f3362d3ce6d78372385711cee0a32178de25e45238c"} Jul 07 14:39:42.119180 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:42.118775 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45780cbd-0927-456b-9745-2cb2e5c8c347","Type":"ContainerStarted","Data":"eb421df710f48cc98cdfbcd6c12a5de8e9a669a9f654a2b4a287fb9a0cc5388b"} Jul 07 14:39:42.154155 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:42.154088 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.47317051 podStartE2EDuration="8.154066948s" totalImagesPullingTime="5.65734096s" totalInitContainerRuntime="23.555478ms" isStatefulPod=false podCreationTimestamp="2026-07-07 14:39:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:39:42.152374497 +0000 UTC m=+323.576560972" watchObservedRunningTime="2026-07-07 14:39:42.154066948 +0000 UTC m=+323.578253424" Jul 07 14:39:43.652397 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:43.652365 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-56fcf86bf-98gk6_f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc/cluster-samples-operator/0.log" Jul 07 14:39:43.659198 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:43.659171 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-56fcf86bf-98gk6_f6d52fde-f231-43d0-9fca-bb2c3bcbf3bc/cluster-samples-operator-watch/0.log" Jul 07 14:39:43.806955 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:43.806921 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8674d957bc-bhcjr"] Jul 07 14:39:44.515847 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:44.515814 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:39:58.185902 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:58.185860 2555 generic.go:356] "Generic (PLEG): container finished" podID="75aeac0c-a9eb-44a9-9ed1-d7412dab8785" containerID="3f6694e89484774b99a984c2cf89a764717566a1812b6f26fcc09edb2e755af1" exitCode=0 Jul 07 14:39:58.186251 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:58.185935 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-b56rb" event={"ID":"75aeac0c-a9eb-44a9-9ed1-d7412dab8785","Type":"ContainerDied","Data":"3f6694e89484774b99a984c2cf89a764717566a1812b6f26fcc09edb2e755af1"} Jul 07 14:39:58.186251 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:58.186218 2555 scope.go:117] "RemoveContainer" containerID="3f6694e89484774b99a984c2cf89a764717566a1812b6f26fcc09edb2e755af1" Jul 07 14:39:59.193490 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:39:59.193445 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-b56rb" event={"ID":"75aeac0c-a9eb-44a9-9ed1-d7412dab8785","Type":"ContainerStarted","Data":"675d97a353c45439d024d543f72566cfa738cbfd34d2ce360a189593a367a114"} Jul 07 14:40:04.213558 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:04.213520 2555 generic.go:356] "Generic (PLEG): container finished" podID="fcd37998-008a-4bf0-b960-bbc6db0dbe76" containerID="8c2061de71408b78ad0e3a1f958137ede085fe02b954a50d1eb85676ea159e24" exitCode=0 Jul 07 14:40:04.214032 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:04.213581 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-vxc5l" event={"ID":"fcd37998-008a-4bf0-b960-bbc6db0dbe76","Type":"ContainerDied","Data":"8c2061de71408b78ad0e3a1f958137ede085fe02b954a50d1eb85676ea159e24"} Jul 07 14:40:04.214032 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:04.213929 2555 scope.go:117] "RemoveContainer" containerID="8c2061de71408b78ad0e3a1f958137ede085fe02b954a50d1eb85676ea159e24" Jul 07 14:40:05.220205 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:05.220164 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-vxc5l" event={"ID":"fcd37998-008a-4bf0-b960-bbc6db0dbe76","Type":"ContainerStarted","Data":"c447e5d394613ec58b0de65a845854a7503ede5b734319ccbae411eb61e5ae8c"} Jul 07 14:40:08.827696 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:08.827655 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-8674d957bc-bhcjr" podUID="63f16300-0736-4975-ad35-2c6a5876338d" containerName="console" containerID="cri-o://667fd77629f7eba05c08c2e838eb13890ce26392ca4443080b6b48766d1fda2e" gracePeriod=15 Jul 07 14:40:09.073740 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:09.073716 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8674d957bc-bhcjr" Jul 07 14:40:09.156429 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:09.156336 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/63f16300-0736-4975-ad35-2c6a5876338d-console-config\") pod \"63f16300-0736-4975-ad35-2c6a5876338d\" (UID: \"63f16300-0736-4975-ad35-2c6a5876338d\") " Jul 07 14:40:09.156429 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:09.156384 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63f16300-0736-4975-ad35-2c6a5876338d-trusted-ca-bundle\") pod \"63f16300-0736-4975-ad35-2c6a5876338d\" (UID: \"63f16300-0736-4975-ad35-2c6a5876338d\") " Jul 07 14:40:09.156429 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:09.156415 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/63f16300-0736-4975-ad35-2c6a5876338d-service-ca\") pod \"63f16300-0736-4975-ad35-2c6a5876338d\" (UID: \"63f16300-0736-4975-ad35-2c6a5876338d\") " Jul 07 14:40:09.156699 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:09.156450 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/63f16300-0736-4975-ad35-2c6a5876338d-oauth-serving-cert\") pod \"63f16300-0736-4975-ad35-2c6a5876338d\" (UID: \"63f16300-0736-4975-ad35-2c6a5876338d\") " Jul 07 14:40:09.156699 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:09.156477 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/63f16300-0736-4975-ad35-2c6a5876338d-console-serving-cert\") pod \"63f16300-0736-4975-ad35-2c6a5876338d\" (UID: \"63f16300-0736-4975-ad35-2c6a5876338d\") " Jul 07 14:40:09.156699 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:09.156570 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/63f16300-0736-4975-ad35-2c6a5876338d-console-oauth-config\") pod \"63f16300-0736-4975-ad35-2c6a5876338d\" (UID: \"63f16300-0736-4975-ad35-2c6a5876338d\") " Jul 07 14:40:09.156699 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:09.156620 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8tdk\" (UniqueName: \"kubernetes.io/projected/63f16300-0736-4975-ad35-2c6a5876338d-kube-api-access-d8tdk\") pod \"63f16300-0736-4975-ad35-2c6a5876338d\" (UID: \"63f16300-0736-4975-ad35-2c6a5876338d\") " Jul 07 14:40:09.156935 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:09.156830 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63f16300-0736-4975-ad35-2c6a5876338d-console-config" (OuterVolumeSpecName: "console-config") pod "63f16300-0736-4975-ad35-2c6a5876338d" (UID: "63f16300-0736-4975-ad35-2c6a5876338d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:40:09.156935 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:09.156876 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63f16300-0736-4975-ad35-2c6a5876338d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "63f16300-0736-4975-ad35-2c6a5876338d" (UID: "63f16300-0736-4975-ad35-2c6a5876338d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:40:09.156935 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:09.156888 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63f16300-0736-4975-ad35-2c6a5876338d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "63f16300-0736-4975-ad35-2c6a5876338d" (UID: "63f16300-0736-4975-ad35-2c6a5876338d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:40:09.156935 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:09.156895 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63f16300-0736-4975-ad35-2c6a5876338d-service-ca" (OuterVolumeSpecName: "service-ca") pod "63f16300-0736-4975-ad35-2c6a5876338d" (UID: "63f16300-0736-4975-ad35-2c6a5876338d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:40:09.157063 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:09.157008 2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/63f16300-0736-4975-ad35-2c6a5876338d-console-config\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:40:09.157063 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:09.157027 2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63f16300-0736-4975-ad35-2c6a5876338d-trusted-ca-bundle\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:40:09.157063 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:09.157042 2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/63f16300-0736-4975-ad35-2c6a5876338d-service-ca\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:40:09.157063 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:09.157055 2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/63f16300-0736-4975-ad35-2c6a5876338d-oauth-serving-cert\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:40:09.158744 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:09.158719 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63f16300-0736-4975-ad35-2c6a5876338d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "63f16300-0736-4975-ad35-2c6a5876338d" (UID: "63f16300-0736-4975-ad35-2c6a5876338d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:40:09.158849 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:09.158798 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63f16300-0736-4975-ad35-2c6a5876338d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "63f16300-0736-4975-ad35-2c6a5876338d" (UID: "63f16300-0736-4975-ad35-2c6a5876338d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:40:09.158964 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:09.158944 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63f16300-0736-4975-ad35-2c6a5876338d-kube-api-access-d8tdk" (OuterVolumeSpecName: "kube-api-access-d8tdk") pod "63f16300-0736-4975-ad35-2c6a5876338d" (UID: "63f16300-0736-4975-ad35-2c6a5876338d"). InnerVolumeSpecName "kube-api-access-d8tdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:40:09.239535 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:09.239504 2555 generic.go:356] "Generic (PLEG): container finished" podID="63f16300-0736-4975-ad35-2c6a5876338d" containerID="667fd77629f7eba05c08c2e838eb13890ce26392ca4443080b6b48766d1fda2e" exitCode=0 Jul 07 14:40:09.239714 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:09.239555 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8674d957bc-bhcjr" Jul 07 14:40:09.239714 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:09.239565 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8674d957bc-bhcjr" event={"ID":"63f16300-0736-4975-ad35-2c6a5876338d","Type":"ContainerDied","Data":"667fd77629f7eba05c08c2e838eb13890ce26392ca4443080b6b48766d1fda2e"} Jul 07 14:40:09.239714 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:09.239610 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8674d957bc-bhcjr" event={"ID":"63f16300-0736-4975-ad35-2c6a5876338d","Type":"ContainerDied","Data":"4d135580218f5f03c482d4736fa15a1b697419373288125beca7af0c91535514"} Jul 07 14:40:09.239714 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:09.239634 2555 scope.go:117] "RemoveContainer" containerID="667fd77629f7eba05c08c2e838eb13890ce26392ca4443080b6b48766d1fda2e" Jul 07 14:40:09.248073 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:09.248057 2555 scope.go:117] "RemoveContainer" containerID="667fd77629f7eba05c08c2e838eb13890ce26392ca4443080b6b48766d1fda2e" Jul 07 14:40:09.248339 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:40:09.248318 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"667fd77629f7eba05c08c2e838eb13890ce26392ca4443080b6b48766d1fda2e\": container with ID starting with 667fd77629f7eba05c08c2e838eb13890ce26392ca4443080b6b48766d1fda2e not found: ID does not exist" containerID="667fd77629f7eba05c08c2e838eb13890ce26392ca4443080b6b48766d1fda2e" Jul 07 14:40:09.248405 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:09.248349 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"667fd77629f7eba05c08c2e838eb13890ce26392ca4443080b6b48766d1fda2e"} err="failed to get container status \"667fd77629f7eba05c08c2e838eb13890ce26392ca4443080b6b48766d1fda2e\": rpc error: code = NotFound desc = could not find container \"667fd77629f7eba05c08c2e838eb13890ce26392ca4443080b6b48766d1fda2e\": container with ID starting with 667fd77629f7eba05c08c2e838eb13890ce26392ca4443080b6b48766d1fda2e not found: ID does not exist" Jul 07 14:40:09.257969 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:09.257948 2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/63f16300-0736-4975-ad35-2c6a5876338d-console-serving-cert\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:40:09.258058 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:09.257970 2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/63f16300-0736-4975-ad35-2c6a5876338d-console-oauth-config\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:40:09.258058 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:09.257983 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8tdk\" (UniqueName: \"kubernetes.io/projected/63f16300-0736-4975-ad35-2c6a5876338d-kube-api-access-d8tdk\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:40:09.261239 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:09.261217 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8674d957bc-bhcjr"] Jul 07 14:40:09.265206 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:09.265187 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8674d957bc-bhcjr"] Jul 07 14:40:11.040878 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:11.040782 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63f16300-0736-4975-ad35-2c6a5876338d" path="/var/lib/kubelet/pods/63f16300-0736-4975-ad35-2c6a5876338d/volumes" Jul 07 14:40:32.847730 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:32.847692 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3bbd20c-73f5-4d24-aed9-145e4b89a590-metrics-certs\") pod \"network-metrics-daemon-lpmtl\" (UID: \"e3bbd20c-73f5-4d24-aed9-145e4b89a590\") " pod="openshift-multus/network-metrics-daemon-lpmtl" Jul 07 14:40:32.850048 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:32.850020 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3bbd20c-73f5-4d24-aed9-145e4b89a590-metrics-certs\") pod \"network-metrics-daemon-lpmtl\" (UID: \"e3bbd20c-73f5-4d24-aed9-145e4b89a590\") " pod="openshift-multus/network-metrics-daemon-lpmtl" Jul 07 14:40:32.940518 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:32.940488 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-v8hh9\"" Jul 07 14:40:32.947911 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:32.947888 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpmtl" Jul 07 14:40:33.080889 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:33.080865 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lpmtl"] Jul 07 14:40:33.083616 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:40:33.083590 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3bbd20c_73f5_4d24_aed9_145e4b89a590.slice/crio-8d3eece10dae6aad6ee878177ad97c43ffca1eca3343d611167dcfcc25fcf701 WatchSource:0}: Error finding container 8d3eece10dae6aad6ee878177ad97c43ffca1eca3343d611167dcfcc25fcf701: Status 404 returned error can't find the container with id 8d3eece10dae6aad6ee878177ad97c43ffca1eca3343d611167dcfcc25fcf701 Jul 07 14:40:33.334048 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:33.334015 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lpmtl" event={"ID":"e3bbd20c-73f5-4d24-aed9-145e4b89a590","Type":"ContainerStarted","Data":"8d3eece10dae6aad6ee878177ad97c43ffca1eca3343d611167dcfcc25fcf701"} Jul 07 14:40:34.339176 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:34.339149 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lpmtl" event={"ID":"e3bbd20c-73f5-4d24-aed9-145e4b89a590","Type":"ContainerStarted","Data":"f206d0ab3d477e822eff5b18a822e0be816aed032cb59dc608b6765d5f47ddfe"} Jul 07 14:40:34.515228 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:34.515136 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:34.530793 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:34.530763 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:35.344260 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:35.344225 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lpmtl" event={"ID":"e3bbd20c-73f5-4d24-aed9-145e4b89a590","Type":"ContainerStarted","Data":"0dba4428d6eed2d8b283b9eb30036988ed8006a1749898f589352675e5458eff"} Jul 07 14:40:35.359645 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:35.359621 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:35.363765 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:35.363704 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lpmtl" podStartSLOduration=374.260447563 podStartE2EDuration="6m15.363690327s" totalImagesPullingTime="1.103242764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:34:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:40:35.362350006 +0000 UTC m=+376.786536481" watchObservedRunningTime="2026-07-07 14:40:35.363690327 +0000 UTC m=+376.787876802" Jul 07 14:40:48.279765 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:48.279716 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 14:40:48.280145 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:48.280043 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="42703fd0-9634-450d-b3a3-0dd97006d417" containerName="alertmanager" containerID="cri-o://4a57cfef091150bd916a79aba28771bcbddbb31c27fd5a32eb7d5ac48b0fe5f5" gracePeriod=120 Jul 07 14:40:48.280145 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:48.280077 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="42703fd0-9634-450d-b3a3-0dd97006d417" containerName="kube-rbac-proxy-metric" containerID="cri-o://90c6641a7740bf6e5762b35cc3f6eed36f190e3316190d0ad8720cd292630682" gracePeriod=120 Jul 07 14:40:48.280271 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:48.280144 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="42703fd0-9634-450d-b3a3-0dd97006d417" containerName="prom-label-proxy" containerID="cri-o://7781de0af30ec08b7ddd50d16c8f1e55428628290cc8e823e867d1c804d561f2" gracePeriod=120 Jul 07 14:40:48.280271 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:48.280140 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="42703fd0-9634-450d-b3a3-0dd97006d417" containerName="kube-rbac-proxy-web" containerID="cri-o://0f22dbc31619d42ab74b8734e1ea2716a378b090c53be7670af12fe15c5cdefc" gracePeriod=120 Jul 07 14:40:48.280271 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:48.280159 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="42703fd0-9634-450d-b3a3-0dd97006d417" containerName="kube-rbac-proxy" containerID="cri-o://b8744c20a7a9b812f69b4282f4a32e74f5fd0d88f2d23d8261db2ca1f82f05a1" gracePeriod=120 Jul 07 14:40:48.280271 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:48.280171 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="42703fd0-9634-450d-b3a3-0dd97006d417" containerName="config-reloader" containerID="cri-o://cbdf6bd18026e5ec68daae1139226f1f4fdea8f9ff58900d86457947eb091bb9" gracePeriod=120 Jul 07 14:40:48.404867 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:48.404841 2555 generic.go:356] "Generic (PLEG): container finished" podID="42703fd0-9634-450d-b3a3-0dd97006d417" containerID="7781de0af30ec08b7ddd50d16c8f1e55428628290cc8e823e867d1c804d561f2" exitCode=0 Jul 07 14:40:48.404867 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:48.404862 2555 generic.go:356] "Generic (PLEG): container finished" podID="42703fd0-9634-450d-b3a3-0dd97006d417" containerID="b8744c20a7a9b812f69b4282f4a32e74f5fd0d88f2d23d8261db2ca1f82f05a1" exitCode=0 Jul 07 14:40:48.404867 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:48.404868 2555 generic.go:356] "Generic (PLEG): container finished" podID="42703fd0-9634-450d-b3a3-0dd97006d417" containerID="cbdf6bd18026e5ec68daae1139226f1f4fdea8f9ff58900d86457947eb091bb9" exitCode=0 Jul 07 14:40:48.404867 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:48.404873 2555 generic.go:356] "Generic (PLEG): container finished" podID="42703fd0-9634-450d-b3a3-0dd97006d417" containerID="4a57cfef091150bd916a79aba28771bcbddbb31c27fd5a32eb7d5ac48b0fe5f5" exitCode=0 Jul 07 14:40:48.405086 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:48.404908 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"42703fd0-9634-450d-b3a3-0dd97006d417","Type":"ContainerDied","Data":"7781de0af30ec08b7ddd50d16c8f1e55428628290cc8e823e867d1c804d561f2"} Jul 07 14:40:48.405086 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:48.404970 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"42703fd0-9634-450d-b3a3-0dd97006d417","Type":"ContainerDied","Data":"b8744c20a7a9b812f69b4282f4a32e74f5fd0d88f2d23d8261db2ca1f82f05a1"} Jul 07 14:40:48.405086 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:48.405017 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"42703fd0-9634-450d-b3a3-0dd97006d417","Type":"ContainerDied","Data":"cbdf6bd18026e5ec68daae1139226f1f4fdea8f9ff58900d86457947eb091bb9"} Jul 07 14:40:48.405086 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:48.405045 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"42703fd0-9634-450d-b3a3-0dd97006d417","Type":"ContainerDied","Data":"4a57cfef091150bd916a79aba28771bcbddbb31c27fd5a32eb7d5ac48b0fe5f5"} Jul 07 14:40:49.413862 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:49.413831 2555 generic.go:356] "Generic (PLEG): container finished" podID="42703fd0-9634-450d-b3a3-0dd97006d417" containerID="90c6641a7740bf6e5762b35cc3f6eed36f190e3316190d0ad8720cd292630682" exitCode=0 Jul 07 14:40:49.413862 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:49.413858 2555 generic.go:356] "Generic (PLEG): container finished" podID="42703fd0-9634-450d-b3a3-0dd97006d417" containerID="0f22dbc31619d42ab74b8734e1ea2716a378b090c53be7670af12fe15c5cdefc" exitCode=0 Jul 07 14:40:49.414229 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:49.413914 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"42703fd0-9634-450d-b3a3-0dd97006d417","Type":"ContainerDied","Data":"90c6641a7740bf6e5762b35cc3f6eed36f190e3316190d0ad8720cd292630682"} Jul 07 14:40:49.414229 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:49.413982 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"42703fd0-9634-450d-b3a3-0dd97006d417","Type":"ContainerDied","Data":"0f22dbc31619d42ab74b8734e1ea2716a378b090c53be7670af12fe15c5cdefc"} Jul 07 14:40:49.527409 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:49.527384 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:40:49.587436 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:49.587407 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42703fd0-9634-450d-b3a3-0dd97006d417-alertmanager-trusted-ca-bundle\") pod \"42703fd0-9634-450d-b3a3-0dd97006d417\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " Jul 07 14:40:49.587436 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:49.587441 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/42703fd0-9634-450d-b3a3-0dd97006d417-secret-alertmanager-kube-rbac-proxy-metric\") pod \"42703fd0-9634-450d-b3a3-0dd97006d417\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " Jul 07 14:40:49.587708 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:49.587461 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/42703fd0-9634-450d-b3a3-0dd97006d417-config-volume\") pod \"42703fd0-9634-450d-b3a3-0dd97006d417\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " Jul 07 14:40:49.587708 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:49.587477 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/42703fd0-9634-450d-b3a3-0dd97006d417-secret-alertmanager-kube-rbac-proxy-web\") pod \"42703fd0-9634-450d-b3a3-0dd97006d417\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " Jul 07 14:40:49.587708 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:49.587544 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/42703fd0-9634-450d-b3a3-0dd97006d417-config-out\") pod \"42703fd0-9634-450d-b3a3-0dd97006d417\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " Jul 07 14:40:49.587708 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:49.587581 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/42703fd0-9634-450d-b3a3-0dd97006d417-tls-assets\") pod \"42703fd0-9634-450d-b3a3-0dd97006d417\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " Jul 07 14:40:49.587708 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:49.587607 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/42703fd0-9634-450d-b3a3-0dd97006d417-secret-alertmanager-kube-rbac-proxy\") pod \"42703fd0-9634-450d-b3a3-0dd97006d417\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " Jul 07 14:40:49.587708 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:49.587700 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/42703fd0-9634-450d-b3a3-0dd97006d417-alertmanager-main-db\") pod \"42703fd0-9634-450d-b3a3-0dd97006d417\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " Jul 07 14:40:49.588211 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:49.587741 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/42703fd0-9634-450d-b3a3-0dd97006d417-cluster-tls-config\") pod \"42703fd0-9634-450d-b3a3-0dd97006d417\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " Jul 07 14:40:49.588211 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:49.587784 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/42703fd0-9634-450d-b3a3-0dd97006d417-web-config\") pod \"42703fd0-9634-450d-b3a3-0dd97006d417\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " Jul 07 14:40:49.588211 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:49.587806 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98lb8\" (UniqueName: \"kubernetes.io/projected/42703fd0-9634-450d-b3a3-0dd97006d417-kube-api-access-98lb8\") pod \"42703fd0-9634-450d-b3a3-0dd97006d417\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " Jul 07 14:40:49.588211 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:49.587838 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/42703fd0-9634-450d-b3a3-0dd97006d417-secret-alertmanager-main-tls\") pod \"42703fd0-9634-450d-b3a3-0dd97006d417\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " Jul 07 14:40:49.588211 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:49.587899 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42703fd0-9634-450d-b3a3-0dd97006d417-metrics-client-ca\") pod \"42703fd0-9634-450d-b3a3-0dd97006d417\" (UID: \"42703fd0-9634-450d-b3a3-0dd97006d417\") " Jul 07 14:40:49.588211 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:49.588020 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42703fd0-9634-450d-b3a3-0dd97006d417-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "42703fd0-9634-450d-b3a3-0dd97006d417" (UID: "42703fd0-9634-450d-b3a3-0dd97006d417"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:40:49.588577 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:49.588377 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42703fd0-9634-450d-b3a3-0dd97006d417-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "42703fd0-9634-450d-b3a3-0dd97006d417" (UID: "42703fd0-9634-450d-b3a3-0dd97006d417"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:40:49.588641 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:49.588576 2555 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42703fd0-9634-450d-b3a3-0dd97006d417-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:40:49.588741 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:49.588715 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42703fd0-9634-450d-b3a3-0dd97006d417-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "42703fd0-9634-450d-b3a3-0dd97006d417" (UID: "42703fd0-9634-450d-b3a3-0dd97006d417"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:40:49.591367 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:49.591150 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42703fd0-9634-450d-b3a3-0dd97006d417-config-volume" (OuterVolumeSpecName: "config-volume") pod "42703fd0-9634-450d-b3a3-0dd97006d417" (UID: "42703fd0-9634-450d-b3a3-0dd97006d417"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:40:49.591367 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:49.591203 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42703fd0-9634-450d-b3a3-0dd97006d417-kube-api-access-98lb8" (OuterVolumeSpecName: "kube-api-access-98lb8") pod "42703fd0-9634-450d-b3a3-0dd97006d417" (UID: "42703fd0-9634-450d-b3a3-0dd97006d417"). InnerVolumeSpecName "kube-api-access-98lb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:40:49.591367 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:49.591209 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42703fd0-9634-450d-b3a3-0dd97006d417-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "42703fd0-9634-450d-b3a3-0dd97006d417" (UID: "42703fd0-9634-450d-b3a3-0dd97006d417"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:40:49.591367 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:49.591232 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42703fd0-9634-450d-b3a3-0dd97006d417-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "42703fd0-9634-450d-b3a3-0dd97006d417" (UID: "42703fd0-9634-450d-b3a3-0dd97006d417"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:40:49.591367 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:49.591322 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42703fd0-9634-450d-b3a3-0dd97006d417-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "42703fd0-9634-450d-b3a3-0dd97006d417" (UID: "42703fd0-9634-450d-b3a3-0dd97006d417"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:40:49.591710 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:49.591497 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42703fd0-9634-450d-b3a3-0dd97006d417-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "42703fd0-9634-450d-b3a3-0dd97006d417" (UID: "42703fd0-9634-450d-b3a3-0dd97006d417"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:40:49.591710 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:49.591586 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42703fd0-9634-450d-b3a3-0dd97006d417-config-out" (OuterVolumeSpecName: "config-out") pod "42703fd0-9634-450d-b3a3-0dd97006d417" (UID: "42703fd0-9634-450d-b3a3-0dd97006d417"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:40:49.592980 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:49.592949 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42703fd0-9634-450d-b3a3-0dd97006d417-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "42703fd0-9634-450d-b3a3-0dd97006d417" (UID: "42703fd0-9634-450d-b3a3-0dd97006d417"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:40:49.597492 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:49.597464 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42703fd0-9634-450d-b3a3-0dd97006d417-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "42703fd0-9634-450d-b3a3-0dd97006d417" (UID: "42703fd0-9634-450d-b3a3-0dd97006d417"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:40:49.603906 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:49.603884 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42703fd0-9634-450d-b3a3-0dd97006d417-web-config" (OuterVolumeSpecName: "web-config") pod "42703fd0-9634-450d-b3a3-0dd97006d417" (UID: "42703fd0-9634-450d-b3a3-0dd97006d417"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:40:49.689939 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:49.689906 2555 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/42703fd0-9634-450d-b3a3-0dd97006d417-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:40:49.689939 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:49.689935 2555 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/42703fd0-9634-450d-b3a3-0dd97006d417-config-volume\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:40:49.689939 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:49.689946 2555 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/42703fd0-9634-450d-b3a3-0dd97006d417-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:40:49.690213 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:49.689954 2555 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/42703fd0-9634-450d-b3a3-0dd97006d417-config-out\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:40:49.690213 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:49.689962 2555 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/42703fd0-9634-450d-b3a3-0dd97006d417-tls-assets\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:40:49.690213 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:49.689970 2555 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/42703fd0-9634-450d-b3a3-0dd97006d417-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:40:49.690213 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:49.689978 2555 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/42703fd0-9634-450d-b3a3-0dd97006d417-alertmanager-main-db\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:40:49.690213 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:49.689986 2555 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/42703fd0-9634-450d-b3a3-0dd97006d417-cluster-tls-config\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:40:49.690213 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:49.689995 2555 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/42703fd0-9634-450d-b3a3-0dd97006d417-web-config\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:40:49.690213 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:49.690002 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98lb8\" (UniqueName: \"kubernetes.io/projected/42703fd0-9634-450d-b3a3-0dd97006d417-kube-api-access-98lb8\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:40:49.690213 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:49.690012 2555 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/42703fd0-9634-450d-b3a3-0dd97006d417-secret-alertmanager-main-tls\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:40:49.690213 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:49.690021 2555 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42703fd0-9634-450d-b3a3-0dd97006d417-metrics-client-ca\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:40:50.420233 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:50.420209 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:40:50.420233 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:50.420220 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"42703fd0-9634-450d-b3a3-0dd97006d417","Type":"ContainerDied","Data":"84122df51d6db342f745ae9c382a9ebda7cd76e4b356fc77bf235803677e6c4e"} Jul 07 14:40:50.420648 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:50.420296 2555 scope.go:117] "RemoveContainer" containerID="7781de0af30ec08b7ddd50d16c8f1e55428628290cc8e823e867d1c804d561f2" Jul 07 14:40:50.429301 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:50.429281 2555 scope.go:117] "RemoveContainer" containerID="90c6641a7740bf6e5762b35cc3f6eed36f190e3316190d0ad8720cd292630682" Jul 07 14:40:50.437341 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:50.437311 2555 scope.go:117] "RemoveContainer" containerID="b8744c20a7a9b812f69b4282f4a32e74f5fd0d88f2d23d8261db2ca1f82f05a1" Jul 07 14:40:50.444178 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:50.444159 2555 scope.go:117] "RemoveContainer" containerID="0f22dbc31619d42ab74b8734e1ea2716a378b090c53be7670af12fe15c5cdefc" Jul 07 14:40:50.447324 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:50.447304 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 14:40:50.452249 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:50.452230 2555 scope.go:117] "RemoveContainer" containerID="cbdf6bd18026e5ec68daae1139226f1f4fdea8f9ff58900d86457947eb091bb9" Jul 07 14:40:50.454004 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:50.453986 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 14:40:50.459102 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:50.459085 2555 scope.go:117] "RemoveContainer" containerID="4a57cfef091150bd916a79aba28771bcbddbb31c27fd5a32eb7d5ac48b0fe5f5" Jul 07 14:40:50.465596 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:50.465577 2555 scope.go:117] "RemoveContainer" containerID="8138979f89d45758b5e8103aa63800e5222211a13f37ac0674104a0ed221c57f" Jul 07 14:40:51.040766 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:51.040725 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42703fd0-9634-450d-b3a3-0dd97006d417" path="/var/lib/kubelet/pods/42703fd0-9634-450d-b3a3-0dd97006d417/volumes" Jul 07 14:40:52.638211 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:52.638177 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 14:40:52.638640 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:52.638465 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="45780cbd-0927-456b-9745-2cb2e5c8c347" containerName="prometheus" containerID="cri-o://8848d48786e07e17317074e50d6d368f2cd9e608751bb9e727d4ad14a49c10a9" gracePeriod=600 Jul 07 14:40:52.638640 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:52.638491 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="45780cbd-0927-456b-9745-2cb2e5c8c347" containerName="kube-rbac-proxy" containerID="cri-o://eb421df710f48cc98cdfbcd6c12a5de8e9a669a9f654a2b4a287fb9a0cc5388b" gracePeriod=600 Jul 07 14:40:52.638640 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:52.638504 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="45780cbd-0927-456b-9745-2cb2e5c8c347" containerName="thanos-sidecar" containerID="cri-o://cf9cd0a46fbc4be92d5225e736bad570b59be29dff591f7e7e2ed032220d0ba7" gracePeriod=600 Jul 07 14:40:52.638640 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:52.638531 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="45780cbd-0927-456b-9745-2cb2e5c8c347" containerName="kube-rbac-proxy-thanos" containerID="cri-o://983a0b90afd6d6be3d792f3362d3ce6d78372385711cee0a32178de25e45238c" gracePeriod=600 Jul 07 14:40:52.638640 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:52.638585 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="45780cbd-0927-456b-9745-2cb2e5c8c347" containerName="kube-rbac-proxy-web" containerID="cri-o://d9831b9e57924d695de6e952962968795c1050a6c862f71e50f9fd13bbfbac9a" gracePeriod=600 Jul 07 14:40:52.638640 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:52.638541 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="45780cbd-0927-456b-9745-2cb2e5c8c347" containerName="config-reloader" containerID="cri-o://5c972dee3f7ea1977c1ea3311e60b942f21feb2596824c17ac897f63ce96c078" gracePeriod=600 Jul 07 14:40:52.881439 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:52.881416 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.015734 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.015663 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45780cbd-0927-456b-9745-2cb2e5c8c347-configmap-metrics-client-ca\") pod \"45780cbd-0927-456b-9745-2cb2e5c8c347\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " Jul 07 14:40:53.015734 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.015693 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/45780cbd-0927-456b-9745-2cb2e5c8c347-config-out\") pod \"45780cbd-0927-456b-9745-2cb2e5c8c347\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " Jul 07 14:40:53.015734 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.015735 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-secret-metrics-client-certs\") pod \"45780cbd-0927-456b-9745-2cb2e5c8c347\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " Jul 07 14:40:53.016031 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.015772 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-secret-kube-rbac-proxy\") pod \"45780cbd-0927-456b-9745-2cb2e5c8c347\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " Jul 07 14:40:53.016031 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.015809 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"45780cbd-0927-456b-9745-2cb2e5c8c347\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " Jul 07 14:40:53.016031 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.015827 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-web-config\") pod \"45780cbd-0927-456b-9745-2cb2e5c8c347\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " Jul 07 14:40:53.016031 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.015854 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/45780cbd-0927-456b-9745-2cb2e5c8c347-prometheus-k8s-rulefiles-2\") pod \"45780cbd-0927-456b-9745-2cb2e5c8c347\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " Jul 07 14:40:53.016031 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.015880 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-secret-grpc-tls\") pod \"45780cbd-0927-456b-9745-2cb2e5c8c347\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " Jul 07 14:40:53.016031 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.015913 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45780cbd-0927-456b-9745-2cb2e5c8c347-configmap-kubelet-serving-ca-bundle\") pod \"45780cbd-0927-456b-9745-2cb2e5c8c347\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " Jul 07 14:40:53.016031 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.015938 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rlw7m\" (UniqueName: \"kubernetes.io/projected/45780cbd-0927-456b-9745-2cb2e5c8c347-kube-api-access-rlw7m\") pod \"45780cbd-0927-456b-9745-2cb2e5c8c347\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " Jul 07 14:40:53.016031 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.015968 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-thanos-prometheus-http-client-file\") pod \"45780cbd-0927-456b-9745-2cb2e5c8c347\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " Jul 07 14:40:53.016031 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.015998 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45780cbd-0927-456b-9745-2cb2e5c8c347-configmap-serving-certs-ca-bundle\") pod \"45780cbd-0927-456b-9745-2cb2e5c8c347\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " Jul 07 14:40:53.016031 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.016030 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45780cbd-0927-456b-9745-2cb2e5c8c347-prometheus-trusted-ca-bundle\") pod \"45780cbd-0927-456b-9745-2cb2e5c8c347\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " Jul 07 14:40:53.016530 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.016060 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-config\") pod \"45780cbd-0927-456b-9745-2cb2e5c8c347\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " Jul 07 14:40:53.016530 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.016087 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-secret-prometheus-k8s-tls\") pod \"45780cbd-0927-456b-9745-2cb2e5c8c347\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " Jul 07 14:40:53.016530 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.016095 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45780cbd-0927-456b-9745-2cb2e5c8c347-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "45780cbd-0927-456b-9745-2cb2e5c8c347" (UID: "45780cbd-0927-456b-9745-2cb2e5c8c347"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:40:53.016530 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.016112 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/45780cbd-0927-456b-9745-2cb2e5c8c347-prometheus-k8s-db\") pod \"45780cbd-0927-456b-9745-2cb2e5c8c347\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " Jul 07 14:40:53.016530 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.016176 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/45780cbd-0927-456b-9745-2cb2e5c8c347-prometheus-k8s-rulefiles-1\") pod \"45780cbd-0927-456b-9745-2cb2e5c8c347\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " Jul 07 14:40:53.016530 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.016209 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/45780cbd-0927-456b-9745-2cb2e5c8c347-tls-assets\") pod \"45780cbd-0927-456b-9745-2cb2e5c8c347\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " Jul 07 14:40:53.016530 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.016239 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"45780cbd-0927-456b-9745-2cb2e5c8c347\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " Jul 07 14:40:53.016530 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.016263 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/45780cbd-0927-456b-9745-2cb2e5c8c347-prometheus-k8s-rulefiles-0\") pod \"45780cbd-0927-456b-9745-2cb2e5c8c347\" (UID: \"45780cbd-0927-456b-9745-2cb2e5c8c347\") " Jul 07 14:40:53.016530 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.016443 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45780cbd-0927-456b-9745-2cb2e5c8c347-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "45780cbd-0927-456b-9745-2cb2e5c8c347" (UID: "45780cbd-0927-456b-9745-2cb2e5c8c347"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:40:53.016986 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.016702 2555 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/45780cbd-0927-456b-9745-2cb2e5c8c347-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:40:53.016986 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.016722 2555 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45780cbd-0927-456b-9745-2cb2e5c8c347-configmap-metrics-client-ca\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:40:53.017293 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.017263 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45780cbd-0927-456b-9745-2cb2e5c8c347-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "45780cbd-0927-456b-9745-2cb2e5c8c347" (UID: "45780cbd-0927-456b-9745-2cb2e5c8c347"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:40:53.017945 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.017913 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45780cbd-0927-456b-9745-2cb2e5c8c347-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "45780cbd-0927-456b-9745-2cb2e5c8c347" (UID: "45780cbd-0927-456b-9745-2cb2e5c8c347"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:40:53.018585 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.018338 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "45780cbd-0927-456b-9745-2cb2e5c8c347" (UID: "45780cbd-0927-456b-9745-2cb2e5c8c347"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:40:53.018585 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.018465 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "45780cbd-0927-456b-9745-2cb2e5c8c347" (UID: "45780cbd-0927-456b-9745-2cb2e5c8c347"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:40:53.018585 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.018531 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45780cbd-0927-456b-9745-2cb2e5c8c347-config-out" (OuterVolumeSpecName: "config-out") pod "45780cbd-0927-456b-9745-2cb2e5c8c347" (UID: "45780cbd-0927-456b-9745-2cb2e5c8c347"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:40:53.018585 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.018559 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "45780cbd-0927-456b-9745-2cb2e5c8c347" (UID: "45780cbd-0927-456b-9745-2cb2e5c8c347"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:40:53.018890 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.018841 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45780cbd-0927-456b-9745-2cb2e5c8c347-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "45780cbd-0927-456b-9745-2cb2e5c8c347" (UID: "45780cbd-0927-456b-9745-2cb2e5c8c347"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:40:53.018890 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.018854 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "45780cbd-0927-456b-9745-2cb2e5c8c347" (UID: "45780cbd-0927-456b-9745-2cb2e5c8c347"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:40:53.019373 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.019125 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45780cbd-0927-456b-9745-2cb2e5c8c347-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "45780cbd-0927-456b-9745-2cb2e5c8c347" (UID: "45780cbd-0927-456b-9745-2cb2e5c8c347"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:40:53.019465 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.019410 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45780cbd-0927-456b-9745-2cb2e5c8c347-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "45780cbd-0927-456b-9745-2cb2e5c8c347" (UID: "45780cbd-0927-456b-9745-2cb2e5c8c347"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:40:53.019727 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.019678 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45780cbd-0927-456b-9745-2cb2e5c8c347-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "45780cbd-0927-456b-9745-2cb2e5c8c347" (UID: "45780cbd-0927-456b-9745-2cb2e5c8c347"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:40:53.019727 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.019690 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45780cbd-0927-456b-9745-2cb2e5c8c347-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "45780cbd-0927-456b-9745-2cb2e5c8c347" (UID: "45780cbd-0927-456b-9745-2cb2e5c8c347"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:40:53.020510 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.020490 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "45780cbd-0927-456b-9745-2cb2e5c8c347" (UID: "45780cbd-0927-456b-9745-2cb2e5c8c347"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:40:53.020647 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.020630 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45780cbd-0927-456b-9745-2cb2e5c8c347-kube-api-access-rlw7m" (OuterVolumeSpecName: "kube-api-access-rlw7m") pod "45780cbd-0927-456b-9745-2cb2e5c8c347" (UID: "45780cbd-0927-456b-9745-2cb2e5c8c347"). InnerVolumeSpecName "kube-api-access-rlw7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:40:53.020922 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.020904 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-config" (OuterVolumeSpecName: "config") pod "45780cbd-0927-456b-9745-2cb2e5c8c347" (UID: "45780cbd-0927-456b-9745-2cb2e5c8c347"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:40:53.021085 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.021067 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "45780cbd-0927-456b-9745-2cb2e5c8c347" (UID: "45780cbd-0927-456b-9745-2cb2e5c8c347"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:40:53.021154 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.021137 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "45780cbd-0927-456b-9745-2cb2e5c8c347" (UID: "45780cbd-0927-456b-9745-2cb2e5c8c347"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:40:53.029185 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.029167 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-web-config" (OuterVolumeSpecName: "web-config") pod "45780cbd-0927-456b-9745-2cb2e5c8c347" (UID: "45780cbd-0927-456b-9745-2cb2e5c8c347"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:40:53.117176 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.117153 2555 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/45780cbd-0927-456b-9745-2cb2e5c8c347-config-out\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:40:53.117176 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.117173 2555 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-secret-metrics-client-certs\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:40:53.117291 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.117184 2555 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-secret-kube-rbac-proxy\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:40:53.117291 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.117198 2555 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:40:53.117291 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.117230 2555 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-web-config\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:40:53.117291 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.117240 2555 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-secret-grpc-tls\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:40:53.117291 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.117248 2555 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45780cbd-0927-456b-9745-2cb2e5c8c347-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:40:53.117291 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.117257 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlw7m\" (UniqueName: \"kubernetes.io/projected/45780cbd-0927-456b-9745-2cb2e5c8c347-kube-api-access-rlw7m\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:40:53.117291 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.117267 2555 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:40:53.117291 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.117280 2555 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45780cbd-0927-456b-9745-2cb2e5c8c347-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:40:53.117291 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.117289 2555 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45780cbd-0927-456b-9745-2cb2e5c8c347-prometheus-trusted-ca-bundle\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:40:53.117564 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.117297 2555 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-config\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:40:53.117564 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.117305 2555 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-secret-prometheus-k8s-tls\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:40:53.117564 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.117314 2555 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/45780cbd-0927-456b-9745-2cb2e5c8c347-prometheus-k8s-db\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:40:53.117564 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.117321 2555 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/45780cbd-0927-456b-9745-2cb2e5c8c347-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:40:53.117564 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.117330 2555 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/45780cbd-0927-456b-9745-2cb2e5c8c347-tls-assets\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:40:53.117564 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.117338 2555 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/45780cbd-0927-456b-9745-2cb2e5c8c347-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:40:53.117564 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.117349 2555 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/45780cbd-0927-456b-9745-2cb2e5c8c347-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:40:53.434654 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.434623 2555 generic.go:356] "Generic (PLEG): container finished" podID="45780cbd-0927-456b-9745-2cb2e5c8c347" containerID="983a0b90afd6d6be3d792f3362d3ce6d78372385711cee0a32178de25e45238c" exitCode=0 Jul 07 14:40:53.434654 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.434646 2555 generic.go:356] "Generic (PLEG): container finished" podID="45780cbd-0927-456b-9745-2cb2e5c8c347" containerID="eb421df710f48cc98cdfbcd6c12a5de8e9a669a9f654a2b4a287fb9a0cc5388b" exitCode=0 Jul 07 14:40:53.434654 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.434652 2555 generic.go:356] "Generic (PLEG): container finished" podID="45780cbd-0927-456b-9745-2cb2e5c8c347" containerID="d9831b9e57924d695de6e952962968795c1050a6c862f71e50f9fd13bbfbac9a" exitCode=0 Jul 07 14:40:53.434654 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.434658 2555 generic.go:356] "Generic (PLEG): container finished" podID="45780cbd-0927-456b-9745-2cb2e5c8c347" containerID="cf9cd0a46fbc4be92d5225e736bad570b59be29dff591f7e7e2ed032220d0ba7" exitCode=0 Jul 07 14:40:53.434654 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.434663 2555 generic.go:356] "Generic (PLEG): container finished" podID="45780cbd-0927-456b-9745-2cb2e5c8c347" containerID="5c972dee3f7ea1977c1ea3311e60b942f21feb2596824c17ac897f63ce96c078" exitCode=0 Jul 07 14:40:53.434957 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.434668 2555 generic.go:356] "Generic (PLEG): container finished" podID="45780cbd-0927-456b-9745-2cb2e5c8c347" containerID="8848d48786e07e17317074e50d6d368f2cd9e608751bb9e727d4ad14a49c10a9" exitCode=0 Jul 07 14:40:53.434957 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.434693 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45780cbd-0927-456b-9745-2cb2e5c8c347","Type":"ContainerDied","Data":"983a0b90afd6d6be3d792f3362d3ce6d78372385711cee0a32178de25e45238c"} Jul 07 14:40:53.434957 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.434705 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.434957 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.434764 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45780cbd-0927-456b-9745-2cb2e5c8c347","Type":"ContainerDied","Data":"eb421df710f48cc98cdfbcd6c12a5de8e9a669a9f654a2b4a287fb9a0cc5388b"} Jul 07 14:40:53.434957 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.434801 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45780cbd-0927-456b-9745-2cb2e5c8c347","Type":"ContainerDied","Data":"d9831b9e57924d695de6e952962968795c1050a6c862f71e50f9fd13bbfbac9a"} Jul 07 14:40:53.434957 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.434826 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45780cbd-0927-456b-9745-2cb2e5c8c347","Type":"ContainerDied","Data":"cf9cd0a46fbc4be92d5225e736bad570b59be29dff591f7e7e2ed032220d0ba7"} Jul 07 14:40:53.434957 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.434848 2555 scope.go:117] "RemoveContainer" containerID="983a0b90afd6d6be3d792f3362d3ce6d78372385711cee0a32178de25e45238c" Jul 07 14:40:53.434957 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.434852 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45780cbd-0927-456b-9745-2cb2e5c8c347","Type":"ContainerDied","Data":"5c972dee3f7ea1977c1ea3311e60b942f21feb2596824c17ac897f63ce96c078"} Jul 07 14:40:53.435272 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.434974 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45780cbd-0927-456b-9745-2cb2e5c8c347","Type":"ContainerDied","Data":"8848d48786e07e17317074e50d6d368f2cd9e608751bb9e727d4ad14a49c10a9"} Jul 07 14:40:53.435272 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.435023 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45780cbd-0927-456b-9745-2cb2e5c8c347","Type":"ContainerDied","Data":"51e478b4b4866bb1d4158c70b06ae08e1cabbbcf69c80ff1f61be211fe8a57e2"} Jul 07 14:40:53.442647 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.442630 2555 scope.go:117] "RemoveContainer" containerID="eb421df710f48cc98cdfbcd6c12a5de8e9a669a9f654a2b4a287fb9a0cc5388b" Jul 07 14:40:53.449417 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.449397 2555 scope.go:117] "RemoveContainer" containerID="d9831b9e57924d695de6e952962968795c1050a6c862f71e50f9fd13bbfbac9a" Jul 07 14:40:53.455973 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.455954 2555 scope.go:117] "RemoveContainer" containerID="cf9cd0a46fbc4be92d5225e736bad570b59be29dff591f7e7e2ed032220d0ba7" Jul 07 14:40:53.457848 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.457826 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 14:40:53.462506 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.462484 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 14:40:53.463547 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.463530 2555 scope.go:117] "RemoveContainer" containerID="5c972dee3f7ea1977c1ea3311e60b942f21feb2596824c17ac897f63ce96c078" Jul 07 14:40:53.470135 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.470120 2555 scope.go:117] "RemoveContainer" containerID="8848d48786e07e17317074e50d6d368f2cd9e608751bb9e727d4ad14a49c10a9" Jul 07 14:40:53.476885 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.476864 2555 scope.go:117] "RemoveContainer" containerID="33a445467b13a1f73e2c3f881972ae29dd6741352f979e8a0121df6ab068ee8c" Jul 07 14:40:53.483179 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.483164 2555 scope.go:117] "RemoveContainer" containerID="983a0b90afd6d6be3d792f3362d3ce6d78372385711cee0a32178de25e45238c" Jul 07 14:40:53.483400 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:40:53.483384 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"983a0b90afd6d6be3d792f3362d3ce6d78372385711cee0a32178de25e45238c\": container with ID starting with 983a0b90afd6d6be3d792f3362d3ce6d78372385711cee0a32178de25e45238c not found: ID does not exist" containerID="983a0b90afd6d6be3d792f3362d3ce6d78372385711cee0a32178de25e45238c" Jul 07 14:40:53.483440 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.483407 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"983a0b90afd6d6be3d792f3362d3ce6d78372385711cee0a32178de25e45238c"} err="failed to get container status \"983a0b90afd6d6be3d792f3362d3ce6d78372385711cee0a32178de25e45238c\": rpc error: code = NotFound desc = could not find container \"983a0b90afd6d6be3d792f3362d3ce6d78372385711cee0a32178de25e45238c\": container with ID starting with 983a0b90afd6d6be3d792f3362d3ce6d78372385711cee0a32178de25e45238c not found: ID does not exist" Jul 07 14:40:53.483440 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.483424 2555 scope.go:117] "RemoveContainer" containerID="eb421df710f48cc98cdfbcd6c12a5de8e9a669a9f654a2b4a287fb9a0cc5388b" Jul 07 14:40:53.483637 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:40:53.483622 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb421df710f48cc98cdfbcd6c12a5de8e9a669a9f654a2b4a287fb9a0cc5388b\": container with ID starting with eb421df710f48cc98cdfbcd6c12a5de8e9a669a9f654a2b4a287fb9a0cc5388b not found: ID does not exist" containerID="eb421df710f48cc98cdfbcd6c12a5de8e9a669a9f654a2b4a287fb9a0cc5388b" Jul 07 14:40:53.483675 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.483640 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb421df710f48cc98cdfbcd6c12a5de8e9a669a9f654a2b4a287fb9a0cc5388b"} err="failed to get container status \"eb421df710f48cc98cdfbcd6c12a5de8e9a669a9f654a2b4a287fb9a0cc5388b\": rpc error: code = NotFound desc = could not find container \"eb421df710f48cc98cdfbcd6c12a5de8e9a669a9f654a2b4a287fb9a0cc5388b\": container with ID starting with eb421df710f48cc98cdfbcd6c12a5de8e9a669a9f654a2b4a287fb9a0cc5388b not found: ID does not exist" Jul 07 14:40:53.483675 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.483652 2555 scope.go:117] "RemoveContainer" containerID="d9831b9e57924d695de6e952962968795c1050a6c862f71e50f9fd13bbfbac9a" Jul 07 14:40:53.483990 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:40:53.483971 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9831b9e57924d695de6e952962968795c1050a6c862f71e50f9fd13bbfbac9a\": container with ID starting with d9831b9e57924d695de6e952962968795c1050a6c862f71e50f9fd13bbfbac9a not found: ID does not exist" containerID="d9831b9e57924d695de6e952962968795c1050a6c862f71e50f9fd13bbfbac9a" Jul 07 14:40:53.484043 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.483996 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9831b9e57924d695de6e952962968795c1050a6c862f71e50f9fd13bbfbac9a"} err="failed to get container status \"d9831b9e57924d695de6e952962968795c1050a6c862f71e50f9fd13bbfbac9a\": rpc error: code = NotFound desc = could not find container \"d9831b9e57924d695de6e952962968795c1050a6c862f71e50f9fd13bbfbac9a\": container with ID starting with d9831b9e57924d695de6e952962968795c1050a6c862f71e50f9fd13bbfbac9a not found: ID does not exist" Jul 07 14:40:53.484043 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.484012 2555 scope.go:117] "RemoveContainer" containerID="cf9cd0a46fbc4be92d5225e736bad570b59be29dff591f7e7e2ed032220d0ba7" Jul 07 14:40:53.484262 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:40:53.484241 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf9cd0a46fbc4be92d5225e736bad570b59be29dff591f7e7e2ed032220d0ba7\": container with ID starting with cf9cd0a46fbc4be92d5225e736bad570b59be29dff591f7e7e2ed032220d0ba7 not found: ID does not exist" containerID="cf9cd0a46fbc4be92d5225e736bad570b59be29dff591f7e7e2ed032220d0ba7" Jul 07 14:40:53.484305 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.484260 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf9cd0a46fbc4be92d5225e736bad570b59be29dff591f7e7e2ed032220d0ba7"} err="failed to get container status \"cf9cd0a46fbc4be92d5225e736bad570b59be29dff591f7e7e2ed032220d0ba7\": rpc error: code = NotFound desc = could not find container \"cf9cd0a46fbc4be92d5225e736bad570b59be29dff591f7e7e2ed032220d0ba7\": container with ID starting with cf9cd0a46fbc4be92d5225e736bad570b59be29dff591f7e7e2ed032220d0ba7 not found: ID does not exist" Jul 07 14:40:53.484305 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.484273 2555 scope.go:117] "RemoveContainer" containerID="5c972dee3f7ea1977c1ea3311e60b942f21feb2596824c17ac897f63ce96c078" Jul 07 14:40:53.484527 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:40:53.484510 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c972dee3f7ea1977c1ea3311e60b942f21feb2596824c17ac897f63ce96c078\": container with ID starting with 5c972dee3f7ea1977c1ea3311e60b942f21feb2596824c17ac897f63ce96c078 not found: ID does not exist" containerID="5c972dee3f7ea1977c1ea3311e60b942f21feb2596824c17ac897f63ce96c078" Jul 07 14:40:53.484587 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.484535 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c972dee3f7ea1977c1ea3311e60b942f21feb2596824c17ac897f63ce96c078"} err="failed to get container status \"5c972dee3f7ea1977c1ea3311e60b942f21feb2596824c17ac897f63ce96c078\": rpc error: code = NotFound desc = could not find container \"5c972dee3f7ea1977c1ea3311e60b942f21feb2596824c17ac897f63ce96c078\": container with ID starting with 5c972dee3f7ea1977c1ea3311e60b942f21feb2596824c17ac897f63ce96c078 not found: ID does not exist" Jul 07 14:40:53.484587 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.484557 2555 scope.go:117] "RemoveContainer" containerID="8848d48786e07e17317074e50d6d368f2cd9e608751bb9e727d4ad14a49c10a9" Jul 07 14:40:53.484786 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:40:53.484764 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8848d48786e07e17317074e50d6d368f2cd9e608751bb9e727d4ad14a49c10a9\": container with ID starting with 8848d48786e07e17317074e50d6d368f2cd9e608751bb9e727d4ad14a49c10a9 not found: ID does not exist" containerID="8848d48786e07e17317074e50d6d368f2cd9e608751bb9e727d4ad14a49c10a9" Jul 07 14:40:53.484831 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.484792 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8848d48786e07e17317074e50d6d368f2cd9e608751bb9e727d4ad14a49c10a9"} err="failed to get container status \"8848d48786e07e17317074e50d6d368f2cd9e608751bb9e727d4ad14a49c10a9\": rpc error: code = NotFound desc = could not find container \"8848d48786e07e17317074e50d6d368f2cd9e608751bb9e727d4ad14a49c10a9\": container with ID starting with 8848d48786e07e17317074e50d6d368f2cd9e608751bb9e727d4ad14a49c10a9 not found: ID does not exist" Jul 07 14:40:53.484831 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.484806 2555 scope.go:117] "RemoveContainer" containerID="33a445467b13a1f73e2c3f881972ae29dd6741352f979e8a0121df6ab068ee8c" Jul 07 14:40:53.485007 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:40:53.484990 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33a445467b13a1f73e2c3f881972ae29dd6741352f979e8a0121df6ab068ee8c\": container with ID starting with 33a445467b13a1f73e2c3f881972ae29dd6741352f979e8a0121df6ab068ee8c not found: ID does not exist" containerID="33a445467b13a1f73e2c3f881972ae29dd6741352f979e8a0121df6ab068ee8c" Jul 07 14:40:53.485072 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.485013 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33a445467b13a1f73e2c3f881972ae29dd6741352f979e8a0121df6ab068ee8c"} err="failed to get container status \"33a445467b13a1f73e2c3f881972ae29dd6741352f979e8a0121df6ab068ee8c\": rpc error: code = NotFound desc = could not find container \"33a445467b13a1f73e2c3f881972ae29dd6741352f979e8a0121df6ab068ee8c\": container with ID starting with 33a445467b13a1f73e2c3f881972ae29dd6741352f979e8a0121df6ab068ee8c not found: ID does not exist" Jul 07 14:40:53.485072 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.485031 2555 scope.go:117] "RemoveContainer" containerID="983a0b90afd6d6be3d792f3362d3ce6d78372385711cee0a32178de25e45238c" Jul 07 14:40:53.485240 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.485225 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"983a0b90afd6d6be3d792f3362d3ce6d78372385711cee0a32178de25e45238c"} err="failed to get container status \"983a0b90afd6d6be3d792f3362d3ce6d78372385711cee0a32178de25e45238c\": rpc error: code = NotFound desc = could not find container \"983a0b90afd6d6be3d792f3362d3ce6d78372385711cee0a32178de25e45238c\": container with ID starting with 983a0b90afd6d6be3d792f3362d3ce6d78372385711cee0a32178de25e45238c not found: ID does not exist" Jul 07 14:40:53.485302 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.485242 2555 scope.go:117] "RemoveContainer" containerID="eb421df710f48cc98cdfbcd6c12a5de8e9a669a9f654a2b4a287fb9a0cc5388b" Jul 07 14:40:53.485489 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.485471 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb421df710f48cc98cdfbcd6c12a5de8e9a669a9f654a2b4a287fb9a0cc5388b"} err="failed to get container status \"eb421df710f48cc98cdfbcd6c12a5de8e9a669a9f654a2b4a287fb9a0cc5388b\": rpc error: code = NotFound desc = could not find container \"eb421df710f48cc98cdfbcd6c12a5de8e9a669a9f654a2b4a287fb9a0cc5388b\": container with ID starting with eb421df710f48cc98cdfbcd6c12a5de8e9a669a9f654a2b4a287fb9a0cc5388b not found: ID does not exist" Jul 07 14:40:53.485528 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.485490 2555 scope.go:117] "RemoveContainer" containerID="d9831b9e57924d695de6e952962968795c1050a6c862f71e50f9fd13bbfbac9a" Jul 07 14:40:53.485715 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.485696 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9831b9e57924d695de6e952962968795c1050a6c862f71e50f9fd13bbfbac9a"} err="failed to get container status \"d9831b9e57924d695de6e952962968795c1050a6c862f71e50f9fd13bbfbac9a\": rpc error: code = NotFound desc = could not find container \"d9831b9e57924d695de6e952962968795c1050a6c862f71e50f9fd13bbfbac9a\": container with ID starting with d9831b9e57924d695de6e952962968795c1050a6c862f71e50f9fd13bbfbac9a not found: ID does not exist" Jul 07 14:40:53.485791 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.485716 2555 scope.go:117] "RemoveContainer" containerID="cf9cd0a46fbc4be92d5225e736bad570b59be29dff591f7e7e2ed032220d0ba7" Jul 07 14:40:53.486244 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.486217 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf9cd0a46fbc4be92d5225e736bad570b59be29dff591f7e7e2ed032220d0ba7"} err="failed to get container status \"cf9cd0a46fbc4be92d5225e736bad570b59be29dff591f7e7e2ed032220d0ba7\": rpc error: code = NotFound desc = could not find container \"cf9cd0a46fbc4be92d5225e736bad570b59be29dff591f7e7e2ed032220d0ba7\": container with ID starting with cf9cd0a46fbc4be92d5225e736bad570b59be29dff591f7e7e2ed032220d0ba7 not found: ID does not exist" Jul 07 14:40:53.486244 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.486245 2555 scope.go:117] "RemoveContainer" containerID="5c972dee3f7ea1977c1ea3311e60b942f21feb2596824c17ac897f63ce96c078" Jul 07 14:40:53.486578 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.486559 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c972dee3f7ea1977c1ea3311e60b942f21feb2596824c17ac897f63ce96c078"} err="failed to get container status \"5c972dee3f7ea1977c1ea3311e60b942f21feb2596824c17ac897f63ce96c078\": rpc error: code = NotFound desc = could not find container \"5c972dee3f7ea1977c1ea3311e60b942f21feb2596824c17ac897f63ce96c078\": container with ID starting with 5c972dee3f7ea1977c1ea3311e60b942f21feb2596824c17ac897f63ce96c078 not found: ID does not exist" Jul 07 14:40:53.486682 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.486579 2555 scope.go:117] "RemoveContainer" containerID="8848d48786e07e17317074e50d6d368f2cd9e608751bb9e727d4ad14a49c10a9" Jul 07 14:40:53.486963 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.486942 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8848d48786e07e17317074e50d6d368f2cd9e608751bb9e727d4ad14a49c10a9"} err="failed to get container status \"8848d48786e07e17317074e50d6d368f2cd9e608751bb9e727d4ad14a49c10a9\": rpc error: code = NotFound desc = could not find container \"8848d48786e07e17317074e50d6d368f2cd9e608751bb9e727d4ad14a49c10a9\": container with ID starting with 8848d48786e07e17317074e50d6d368f2cd9e608751bb9e727d4ad14a49c10a9 not found: ID does not exist" Jul 07 14:40:53.486963 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.486962 2555 scope.go:117] "RemoveContainer" containerID="33a445467b13a1f73e2c3f881972ae29dd6741352f979e8a0121df6ab068ee8c" Jul 07 14:40:53.487226 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.487204 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33a445467b13a1f73e2c3f881972ae29dd6741352f979e8a0121df6ab068ee8c"} err="failed to get container status \"33a445467b13a1f73e2c3f881972ae29dd6741352f979e8a0121df6ab068ee8c\": rpc error: code = NotFound desc = could not find container \"33a445467b13a1f73e2c3f881972ae29dd6741352f979e8a0121df6ab068ee8c\": container with ID starting with 33a445467b13a1f73e2c3f881972ae29dd6741352f979e8a0121df6ab068ee8c not found: ID does not exist" Jul 07 14:40:53.487226 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.487226 2555 scope.go:117] "RemoveContainer" containerID="983a0b90afd6d6be3d792f3362d3ce6d78372385711cee0a32178de25e45238c" Jul 07 14:40:53.487470 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.487454 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"983a0b90afd6d6be3d792f3362d3ce6d78372385711cee0a32178de25e45238c"} err="failed to get container status \"983a0b90afd6d6be3d792f3362d3ce6d78372385711cee0a32178de25e45238c\": rpc error: code = NotFound desc = could not find container \"983a0b90afd6d6be3d792f3362d3ce6d78372385711cee0a32178de25e45238c\": container with ID starting with 983a0b90afd6d6be3d792f3362d3ce6d78372385711cee0a32178de25e45238c not found: ID does not exist" Jul 07 14:40:53.487569 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.487471 2555 scope.go:117] "RemoveContainer" containerID="eb421df710f48cc98cdfbcd6c12a5de8e9a669a9f654a2b4a287fb9a0cc5388b" Jul 07 14:40:53.487705 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.487688 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb421df710f48cc98cdfbcd6c12a5de8e9a669a9f654a2b4a287fb9a0cc5388b"} err="failed to get container status \"eb421df710f48cc98cdfbcd6c12a5de8e9a669a9f654a2b4a287fb9a0cc5388b\": rpc error: code = NotFound desc = could not find container \"eb421df710f48cc98cdfbcd6c12a5de8e9a669a9f654a2b4a287fb9a0cc5388b\": container with ID starting with eb421df710f48cc98cdfbcd6c12a5de8e9a669a9f654a2b4a287fb9a0cc5388b not found: ID does not exist" Jul 07 14:40:53.487774 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.487705 2555 scope.go:117] "RemoveContainer" containerID="d9831b9e57924d695de6e952962968795c1050a6c862f71e50f9fd13bbfbac9a" Jul 07 14:40:53.487774 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.487740 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 14:40:53.487940 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.487923 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9831b9e57924d695de6e952962968795c1050a6c862f71e50f9fd13bbfbac9a"} err="failed to get container status \"d9831b9e57924d695de6e952962968795c1050a6c862f71e50f9fd13bbfbac9a\": rpc error: code = NotFound desc = could not find container \"d9831b9e57924d695de6e952962968795c1050a6c862f71e50f9fd13bbfbac9a\": container with ID starting with d9831b9e57924d695de6e952962968795c1050a6c862f71e50f9fd13bbfbac9a not found: ID does not exist" Jul 07 14:40:53.487940 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.487940 2555 scope.go:117] "RemoveContainer" containerID="cf9cd0a46fbc4be92d5225e736bad570b59be29dff591f7e7e2ed032220d0ba7" Jul 07 14:40:53.488026 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488018 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45780cbd-0927-456b-9745-2cb2e5c8c347" containerName="thanos-sidecar" Jul 07 14:40:53.488074 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488031 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="45780cbd-0927-456b-9745-2cb2e5c8c347" containerName="thanos-sidecar" Jul 07 14:40:53.488074 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488039 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45780cbd-0927-456b-9745-2cb2e5c8c347" containerName="kube-rbac-proxy" Jul 07 14:40:53.488074 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488045 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="45780cbd-0927-456b-9745-2cb2e5c8c347" containerName="kube-rbac-proxy" Jul 07 14:40:53.488074 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488056 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45780cbd-0927-456b-9745-2cb2e5c8c347" containerName="prometheus" Jul 07 14:40:53.488074 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488064 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="45780cbd-0927-456b-9745-2cb2e5c8c347" containerName="prometheus" Jul 07 14:40:53.488074 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488072 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45780cbd-0927-456b-9745-2cb2e5c8c347" containerName="kube-rbac-proxy-thanos" Jul 07 14:40:53.488358 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488082 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="45780cbd-0927-456b-9745-2cb2e5c8c347" containerName="kube-rbac-proxy-thanos" Jul 07 14:40:53.488358 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488093 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42703fd0-9634-450d-b3a3-0dd97006d417" containerName="init-config-reloader" Jul 07 14:40:53.488358 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488101 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="42703fd0-9634-450d-b3a3-0dd97006d417" containerName="init-config-reloader" Jul 07 14:40:53.488358 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488110 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42703fd0-9634-450d-b3a3-0dd97006d417" containerName="alertmanager" Jul 07 14:40:53.488358 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488115 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="42703fd0-9634-450d-b3a3-0dd97006d417" containerName="alertmanager" Jul 07 14:40:53.488358 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488123 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63f16300-0736-4975-ad35-2c6a5876338d" containerName="console" Jul 07 14:40:53.488358 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488128 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="63f16300-0736-4975-ad35-2c6a5876338d" containerName="console" Jul 07 14:40:53.488358 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488135 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45780cbd-0927-456b-9745-2cb2e5c8c347" containerName="init-config-reloader" Jul 07 14:40:53.488358 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488138 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf9cd0a46fbc4be92d5225e736bad570b59be29dff591f7e7e2ed032220d0ba7"} err="failed to get container status \"cf9cd0a46fbc4be92d5225e736bad570b59be29dff591f7e7e2ed032220d0ba7\": rpc error: code = NotFound desc = could not find container \"cf9cd0a46fbc4be92d5225e736bad570b59be29dff591f7e7e2ed032220d0ba7\": container with ID starting with cf9cd0a46fbc4be92d5225e736bad570b59be29dff591f7e7e2ed032220d0ba7 not found: ID does not exist" Jul 07 14:40:53.488358 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488140 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="45780cbd-0927-456b-9745-2cb2e5c8c347" containerName="init-config-reloader" Jul 07 14:40:53.488358 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488162 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42703fd0-9634-450d-b3a3-0dd97006d417" containerName="prom-label-proxy" Jul 07 14:40:53.488358 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488161 2555 scope.go:117] "RemoveContainer" containerID="5c972dee3f7ea1977c1ea3311e60b942f21feb2596824c17ac897f63ce96c078" Jul 07 14:40:53.488358 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488167 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="42703fd0-9634-450d-b3a3-0dd97006d417" containerName="prom-label-proxy" Jul 07 14:40:53.488358 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488229 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42703fd0-9634-450d-b3a3-0dd97006d417" containerName="config-reloader" Jul 07 14:40:53.488358 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488242 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="42703fd0-9634-450d-b3a3-0dd97006d417" containerName="config-reloader" Jul 07 14:40:53.488358 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488256 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45780cbd-0927-456b-9745-2cb2e5c8c347" containerName="config-reloader" Jul 07 14:40:53.488358 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488264 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="45780cbd-0927-456b-9745-2cb2e5c8c347" containerName="config-reloader" Jul 07 14:40:53.488358 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488274 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42703fd0-9634-450d-b3a3-0dd97006d417" containerName="kube-rbac-proxy-metric" Jul 07 14:40:53.488358 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488283 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="42703fd0-9634-450d-b3a3-0dd97006d417" containerName="kube-rbac-proxy-metric" Jul 07 14:40:53.488358 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488296 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45780cbd-0927-456b-9745-2cb2e5c8c347" containerName="kube-rbac-proxy-web" Jul 07 14:40:53.488358 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488304 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="45780cbd-0927-456b-9745-2cb2e5c8c347" containerName="kube-rbac-proxy-web" Jul 07 14:40:53.488358 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488325 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42703fd0-9634-450d-b3a3-0dd97006d417" containerName="kube-rbac-proxy-web" Jul 07 14:40:53.488358 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488333 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="42703fd0-9634-450d-b3a3-0dd97006d417" containerName="kube-rbac-proxy-web" Jul 07 14:40:53.488358 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488346 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42703fd0-9634-450d-b3a3-0dd97006d417" containerName="kube-rbac-proxy" Jul 07 14:40:53.488358 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488354 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="42703fd0-9634-450d-b3a3-0dd97006d417" containerName="kube-rbac-proxy" Jul 07 14:40:53.489678 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488379 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c972dee3f7ea1977c1ea3311e60b942f21feb2596824c17ac897f63ce96c078"} err="failed to get container status \"5c972dee3f7ea1977c1ea3311e60b942f21feb2596824c17ac897f63ce96c078\": rpc error: code = NotFound desc = could not find container \"5c972dee3f7ea1977c1ea3311e60b942f21feb2596824c17ac897f63ce96c078\": container with ID starting with 5c972dee3f7ea1977c1ea3311e60b942f21feb2596824c17ac897f63ce96c078 not found: ID does not exist" Jul 07 14:40:53.489678 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488394 2555 scope.go:117] "RemoveContainer" containerID="8848d48786e07e17317074e50d6d368f2cd9e608751bb9e727d4ad14a49c10a9" Jul 07 14:40:53.489678 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488439 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="45780cbd-0927-456b-9745-2cb2e5c8c347" containerName="config-reloader" Jul 07 14:40:53.489678 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488454 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="45780cbd-0927-456b-9745-2cb2e5c8c347" containerName="kube-rbac-proxy-web" Jul 07 14:40:53.489678 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488463 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="42703fd0-9634-450d-b3a3-0dd97006d417" containerName="alertmanager" Jul 07 14:40:53.489678 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488473 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="42703fd0-9634-450d-b3a3-0dd97006d417" containerName="kube-rbac-proxy" Jul 07 14:40:53.489678 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488483 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="45780cbd-0927-456b-9745-2cb2e5c8c347" containerName="thanos-sidecar" Jul 07 14:40:53.489678 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488491 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="42703fd0-9634-450d-b3a3-0dd97006d417" containerName="prom-label-proxy" Jul 07 14:40:53.489678 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488500 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="42703fd0-9634-450d-b3a3-0dd97006d417" containerName="kube-rbac-proxy-metric" Jul 07 14:40:53.489678 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488507 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="45780cbd-0927-456b-9745-2cb2e5c8c347" containerName="kube-rbac-proxy" Jul 07 14:40:53.489678 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488517 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="42703fd0-9634-450d-b3a3-0dd97006d417" containerName="config-reloader" Jul 07 14:40:53.489678 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488526 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="63f16300-0736-4975-ad35-2c6a5876338d" containerName="console" Jul 07 14:40:53.489678 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488534 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="45780cbd-0927-456b-9745-2cb2e5c8c347" containerName="prometheus" Jul 07 14:40:53.489678 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488543 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="42703fd0-9634-450d-b3a3-0dd97006d417" containerName="kube-rbac-proxy-web" Jul 07 14:40:53.489678 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488551 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="45780cbd-0927-456b-9745-2cb2e5c8c347" containerName="kube-rbac-proxy-thanos" Jul 07 14:40:53.489678 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488625 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8848d48786e07e17317074e50d6d368f2cd9e608751bb9e727d4ad14a49c10a9"} err="failed to get container status \"8848d48786e07e17317074e50d6d368f2cd9e608751bb9e727d4ad14a49c10a9\": rpc error: code = NotFound desc = could not find container \"8848d48786e07e17317074e50d6d368f2cd9e608751bb9e727d4ad14a49c10a9\": container with ID starting with 8848d48786e07e17317074e50d6d368f2cd9e608751bb9e727d4ad14a49c10a9 not found: ID does not exist" Jul 07 14:40:53.489678 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488639 2555 scope.go:117] "RemoveContainer" containerID="33a445467b13a1f73e2c3f881972ae29dd6741352f979e8a0121df6ab068ee8c" Jul 07 14:40:53.489678 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488876 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33a445467b13a1f73e2c3f881972ae29dd6741352f979e8a0121df6ab068ee8c"} err="failed to get container status \"33a445467b13a1f73e2c3f881972ae29dd6741352f979e8a0121df6ab068ee8c\": rpc error: code = NotFound desc = could not find container \"33a445467b13a1f73e2c3f881972ae29dd6741352f979e8a0121df6ab068ee8c\": container with ID starting with 33a445467b13a1f73e2c3f881972ae29dd6741352f979e8a0121df6ab068ee8c not found: ID does not exist" Jul 07 14:40:53.489678 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.488895 2555 scope.go:117] "RemoveContainer" containerID="983a0b90afd6d6be3d792f3362d3ce6d78372385711cee0a32178de25e45238c" Jul 07 14:40:53.489678 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.489136 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"983a0b90afd6d6be3d792f3362d3ce6d78372385711cee0a32178de25e45238c"} err="failed to get container status \"983a0b90afd6d6be3d792f3362d3ce6d78372385711cee0a32178de25e45238c\": rpc error: code = NotFound desc = could not find container \"983a0b90afd6d6be3d792f3362d3ce6d78372385711cee0a32178de25e45238c\": container with ID starting with 983a0b90afd6d6be3d792f3362d3ce6d78372385711cee0a32178de25e45238c not found: ID does not exist" Jul 07 14:40:53.489678 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.489159 2555 scope.go:117] "RemoveContainer" containerID="eb421df710f48cc98cdfbcd6c12a5de8e9a669a9f654a2b4a287fb9a0cc5388b" Jul 07 14:40:53.489678 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.489356 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb421df710f48cc98cdfbcd6c12a5de8e9a669a9f654a2b4a287fb9a0cc5388b"} err="failed to get container status \"eb421df710f48cc98cdfbcd6c12a5de8e9a669a9f654a2b4a287fb9a0cc5388b\": rpc error: code = NotFound desc = could not find container \"eb421df710f48cc98cdfbcd6c12a5de8e9a669a9f654a2b4a287fb9a0cc5388b\": container with ID starting with eb421df710f48cc98cdfbcd6c12a5de8e9a669a9f654a2b4a287fb9a0cc5388b not found: ID does not exist" Jul 07 14:40:53.489678 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.489371 2555 scope.go:117] "RemoveContainer" containerID="d9831b9e57924d695de6e952962968795c1050a6c862f71e50f9fd13bbfbac9a" Jul 07 14:40:53.490520 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.489560 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9831b9e57924d695de6e952962968795c1050a6c862f71e50f9fd13bbfbac9a"} err="failed to get container status \"d9831b9e57924d695de6e952962968795c1050a6c862f71e50f9fd13bbfbac9a\": rpc error: code = NotFound desc = could not find container \"d9831b9e57924d695de6e952962968795c1050a6c862f71e50f9fd13bbfbac9a\": container with ID starting with d9831b9e57924d695de6e952962968795c1050a6c862f71e50f9fd13bbfbac9a not found: ID does not exist" Jul 07 14:40:53.490520 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.489579 2555 scope.go:117] "RemoveContainer" containerID="cf9cd0a46fbc4be92d5225e736bad570b59be29dff591f7e7e2ed032220d0ba7" Jul 07 14:40:53.490520 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.489808 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf9cd0a46fbc4be92d5225e736bad570b59be29dff591f7e7e2ed032220d0ba7"} err="failed to get container status \"cf9cd0a46fbc4be92d5225e736bad570b59be29dff591f7e7e2ed032220d0ba7\": rpc error: code = NotFound desc = could not find container \"cf9cd0a46fbc4be92d5225e736bad570b59be29dff591f7e7e2ed032220d0ba7\": container with ID starting with cf9cd0a46fbc4be92d5225e736bad570b59be29dff591f7e7e2ed032220d0ba7 not found: ID does not exist" Jul 07 14:40:53.490520 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.489824 2555 scope.go:117] "RemoveContainer" containerID="5c972dee3f7ea1977c1ea3311e60b942f21feb2596824c17ac897f63ce96c078" Jul 07 14:40:53.490520 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.490039 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c972dee3f7ea1977c1ea3311e60b942f21feb2596824c17ac897f63ce96c078"} err="failed to get container status \"5c972dee3f7ea1977c1ea3311e60b942f21feb2596824c17ac897f63ce96c078\": rpc error: code = NotFound desc = could not find container \"5c972dee3f7ea1977c1ea3311e60b942f21feb2596824c17ac897f63ce96c078\": container with ID starting with 5c972dee3f7ea1977c1ea3311e60b942f21feb2596824c17ac897f63ce96c078 not found: ID does not exist" Jul 07 14:40:53.490520 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.490053 2555 scope.go:117] "RemoveContainer" containerID="8848d48786e07e17317074e50d6d368f2cd9e608751bb9e727d4ad14a49c10a9" Jul 07 14:40:53.490520 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.490250 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8848d48786e07e17317074e50d6d368f2cd9e608751bb9e727d4ad14a49c10a9"} err="failed to get container status \"8848d48786e07e17317074e50d6d368f2cd9e608751bb9e727d4ad14a49c10a9\": rpc error: code = NotFound desc = could not find container \"8848d48786e07e17317074e50d6d368f2cd9e608751bb9e727d4ad14a49c10a9\": container with ID starting with 8848d48786e07e17317074e50d6d368f2cd9e608751bb9e727d4ad14a49c10a9 not found: ID does not exist" Jul 07 14:40:53.490520 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.490266 2555 scope.go:117] "RemoveContainer" containerID="33a445467b13a1f73e2c3f881972ae29dd6741352f979e8a0121df6ab068ee8c" Jul 07 14:40:53.490520 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.490454 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33a445467b13a1f73e2c3f881972ae29dd6741352f979e8a0121df6ab068ee8c"} err="failed to get container status \"33a445467b13a1f73e2c3f881972ae29dd6741352f979e8a0121df6ab068ee8c\": rpc error: code = NotFound desc = could not find container \"33a445467b13a1f73e2c3f881972ae29dd6741352f979e8a0121df6ab068ee8c\": container with ID starting with 33a445467b13a1f73e2c3f881972ae29dd6741352f979e8a0121df6ab068ee8c not found: ID does not exist" Jul 07 14:40:53.490520 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.490468 2555 scope.go:117] "RemoveContainer" containerID="983a0b90afd6d6be3d792f3362d3ce6d78372385711cee0a32178de25e45238c" Jul 07 14:40:53.490907 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.490677 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"983a0b90afd6d6be3d792f3362d3ce6d78372385711cee0a32178de25e45238c"} err="failed to get container status \"983a0b90afd6d6be3d792f3362d3ce6d78372385711cee0a32178de25e45238c\": rpc error: code = NotFound desc = could not find container \"983a0b90afd6d6be3d792f3362d3ce6d78372385711cee0a32178de25e45238c\": container with ID starting with 983a0b90afd6d6be3d792f3362d3ce6d78372385711cee0a32178de25e45238c not found: ID does not exist" Jul 07 14:40:53.490907 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.490692 2555 scope.go:117] "RemoveContainer" containerID="eb421df710f48cc98cdfbcd6c12a5de8e9a669a9f654a2b4a287fb9a0cc5388b" Jul 07 14:40:53.490907 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.490870 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb421df710f48cc98cdfbcd6c12a5de8e9a669a9f654a2b4a287fb9a0cc5388b"} err="failed to get container status \"eb421df710f48cc98cdfbcd6c12a5de8e9a669a9f654a2b4a287fb9a0cc5388b\": rpc error: code = NotFound desc = could not find container \"eb421df710f48cc98cdfbcd6c12a5de8e9a669a9f654a2b4a287fb9a0cc5388b\": container with ID starting with eb421df710f48cc98cdfbcd6c12a5de8e9a669a9f654a2b4a287fb9a0cc5388b not found: ID does not exist" Jul 07 14:40:53.490907 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.490882 2555 scope.go:117] "RemoveContainer" containerID="d9831b9e57924d695de6e952962968795c1050a6c862f71e50f9fd13bbfbac9a" Jul 07 14:40:53.491060 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.491027 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9831b9e57924d695de6e952962968795c1050a6c862f71e50f9fd13bbfbac9a"} err="failed to get container status \"d9831b9e57924d695de6e952962968795c1050a6c862f71e50f9fd13bbfbac9a\": rpc error: code = NotFound desc = could not find container \"d9831b9e57924d695de6e952962968795c1050a6c862f71e50f9fd13bbfbac9a\": container with ID starting with d9831b9e57924d695de6e952962968795c1050a6c862f71e50f9fd13bbfbac9a not found: ID does not exist" Jul 07 14:40:53.491060 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.491038 2555 scope.go:117] "RemoveContainer" containerID="cf9cd0a46fbc4be92d5225e736bad570b59be29dff591f7e7e2ed032220d0ba7" Jul 07 14:40:53.491235 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.491216 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf9cd0a46fbc4be92d5225e736bad570b59be29dff591f7e7e2ed032220d0ba7"} err="failed to get container status \"cf9cd0a46fbc4be92d5225e736bad570b59be29dff591f7e7e2ed032220d0ba7\": rpc error: code = NotFound desc = could not find container \"cf9cd0a46fbc4be92d5225e736bad570b59be29dff591f7e7e2ed032220d0ba7\": container with ID starting with cf9cd0a46fbc4be92d5225e736bad570b59be29dff591f7e7e2ed032220d0ba7 not found: ID does not exist" Jul 07 14:40:53.491277 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.491234 2555 scope.go:117] "RemoveContainer" containerID="5c972dee3f7ea1977c1ea3311e60b942f21feb2596824c17ac897f63ce96c078" Jul 07 14:40:53.491419 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.491401 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c972dee3f7ea1977c1ea3311e60b942f21feb2596824c17ac897f63ce96c078"} err="failed to get container status \"5c972dee3f7ea1977c1ea3311e60b942f21feb2596824c17ac897f63ce96c078\": rpc error: code = NotFound desc = could not find container \"5c972dee3f7ea1977c1ea3311e60b942f21feb2596824c17ac897f63ce96c078\": container with ID starting with 5c972dee3f7ea1977c1ea3311e60b942f21feb2596824c17ac897f63ce96c078 not found: ID does not exist" Jul 07 14:40:53.491484 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.491421 2555 scope.go:117] "RemoveContainer" containerID="8848d48786e07e17317074e50d6d368f2cd9e608751bb9e727d4ad14a49c10a9" Jul 07 14:40:53.491613 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.491596 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8848d48786e07e17317074e50d6d368f2cd9e608751bb9e727d4ad14a49c10a9"} err="failed to get container status \"8848d48786e07e17317074e50d6d368f2cd9e608751bb9e727d4ad14a49c10a9\": rpc error: code = NotFound desc = could not find container \"8848d48786e07e17317074e50d6d368f2cd9e608751bb9e727d4ad14a49c10a9\": container with ID starting with 8848d48786e07e17317074e50d6d368f2cd9e608751bb9e727d4ad14a49c10a9 not found: ID does not exist" Jul 07 14:40:53.491660 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.491614 2555 scope.go:117] "RemoveContainer" containerID="33a445467b13a1f73e2c3f881972ae29dd6741352f979e8a0121df6ab068ee8c" Jul 07 14:40:53.491809 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.491789 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33a445467b13a1f73e2c3f881972ae29dd6741352f979e8a0121df6ab068ee8c"} err="failed to get container status \"33a445467b13a1f73e2c3f881972ae29dd6741352f979e8a0121df6ab068ee8c\": rpc error: code = NotFound desc = could not find container \"33a445467b13a1f73e2c3f881972ae29dd6741352f979e8a0121df6ab068ee8c\": container with ID starting with 33a445467b13a1f73e2c3f881972ae29dd6741352f979e8a0121df6ab068ee8c not found: ID does not exist" Jul 07 14:40:53.491856 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.491810 2555 scope.go:117] "RemoveContainer" containerID="983a0b90afd6d6be3d792f3362d3ce6d78372385711cee0a32178de25e45238c" Jul 07 14:40:53.491987 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.491971 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"983a0b90afd6d6be3d792f3362d3ce6d78372385711cee0a32178de25e45238c"} err="failed to get container status \"983a0b90afd6d6be3d792f3362d3ce6d78372385711cee0a32178de25e45238c\": rpc error: code = NotFound desc = could not find container \"983a0b90afd6d6be3d792f3362d3ce6d78372385711cee0a32178de25e45238c\": container with ID starting with 983a0b90afd6d6be3d792f3362d3ce6d78372385711cee0a32178de25e45238c not found: ID does not exist" Jul 07 14:40:53.492036 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.491988 2555 scope.go:117] "RemoveContainer" containerID="eb421df710f48cc98cdfbcd6c12a5de8e9a669a9f654a2b4a287fb9a0cc5388b" Jul 07 14:40:53.492171 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.492156 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb421df710f48cc98cdfbcd6c12a5de8e9a669a9f654a2b4a287fb9a0cc5388b"} err="failed to get container status \"eb421df710f48cc98cdfbcd6c12a5de8e9a669a9f654a2b4a287fb9a0cc5388b\": rpc error: code = NotFound desc = could not find container \"eb421df710f48cc98cdfbcd6c12a5de8e9a669a9f654a2b4a287fb9a0cc5388b\": container with ID starting with eb421df710f48cc98cdfbcd6c12a5de8e9a669a9f654a2b4a287fb9a0cc5388b not found: ID does not exist" Jul 07 14:40:53.492221 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.492170 2555 scope.go:117] "RemoveContainer" containerID="d9831b9e57924d695de6e952962968795c1050a6c862f71e50f9fd13bbfbac9a" Jul 07 14:40:53.492414 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.492397 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9831b9e57924d695de6e952962968795c1050a6c862f71e50f9fd13bbfbac9a"} err="failed to get container status \"d9831b9e57924d695de6e952962968795c1050a6c862f71e50f9fd13bbfbac9a\": rpc error: code = NotFound desc = could not find container \"d9831b9e57924d695de6e952962968795c1050a6c862f71e50f9fd13bbfbac9a\": container with ID starting with d9831b9e57924d695de6e952962968795c1050a6c862f71e50f9fd13bbfbac9a not found: ID does not exist" Jul 07 14:40:53.492462 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.492413 2555 scope.go:117] "RemoveContainer" containerID="cf9cd0a46fbc4be92d5225e736bad570b59be29dff591f7e7e2ed032220d0ba7" Jul 07 14:40:53.492599 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.492583 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf9cd0a46fbc4be92d5225e736bad570b59be29dff591f7e7e2ed032220d0ba7"} err="failed to get container status \"cf9cd0a46fbc4be92d5225e736bad570b59be29dff591f7e7e2ed032220d0ba7\": rpc error: code = NotFound desc = could not find container \"cf9cd0a46fbc4be92d5225e736bad570b59be29dff591f7e7e2ed032220d0ba7\": container with ID starting with cf9cd0a46fbc4be92d5225e736bad570b59be29dff591f7e7e2ed032220d0ba7 not found: ID does not exist" Jul 07 14:40:53.492640 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.492600 2555 scope.go:117] "RemoveContainer" containerID="5c972dee3f7ea1977c1ea3311e60b942f21feb2596824c17ac897f63ce96c078" Jul 07 14:40:53.492794 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.492777 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c972dee3f7ea1977c1ea3311e60b942f21feb2596824c17ac897f63ce96c078"} err="failed to get container status \"5c972dee3f7ea1977c1ea3311e60b942f21feb2596824c17ac897f63ce96c078\": rpc error: code = NotFound desc = could not find container \"5c972dee3f7ea1977c1ea3311e60b942f21feb2596824c17ac897f63ce96c078\": container with ID starting with 5c972dee3f7ea1977c1ea3311e60b942f21feb2596824c17ac897f63ce96c078 not found: ID does not exist" Jul 07 14:40:53.492861 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.492794 2555 scope.go:117] "RemoveContainer" containerID="8848d48786e07e17317074e50d6d368f2cd9e608751bb9e727d4ad14a49c10a9" Jul 07 14:40:53.492974 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.492956 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8848d48786e07e17317074e50d6d368f2cd9e608751bb9e727d4ad14a49c10a9"} err="failed to get container status \"8848d48786e07e17317074e50d6d368f2cd9e608751bb9e727d4ad14a49c10a9\": rpc error: code = NotFound desc = could not find container \"8848d48786e07e17317074e50d6d368f2cd9e608751bb9e727d4ad14a49c10a9\": container with ID starting with 8848d48786e07e17317074e50d6d368f2cd9e608751bb9e727d4ad14a49c10a9 not found: ID does not exist" Jul 07 14:40:53.493008 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.492976 2555 scope.go:117] "RemoveContainer" containerID="33a445467b13a1f73e2c3f881972ae29dd6741352f979e8a0121df6ab068ee8c" Jul 07 14:40:53.493154 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.493140 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33a445467b13a1f73e2c3f881972ae29dd6741352f979e8a0121df6ab068ee8c"} err="failed to get container status \"33a445467b13a1f73e2c3f881972ae29dd6741352f979e8a0121df6ab068ee8c\": rpc error: code = NotFound desc = could not find container \"33a445467b13a1f73e2c3f881972ae29dd6741352f979e8a0121df6ab068ee8c\": container with ID starting with 33a445467b13a1f73e2c3f881972ae29dd6741352f979e8a0121df6ab068ee8c not found: ID does not exist" Jul 07 14:40:53.494196 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.494177 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.496934 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.496914 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 07 14:40:53.496934 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.496916 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 07 14:40:53.498475 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.497434 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 07 14:40:53.498475 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.497453 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 07 14:40:53.498475 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.497548 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 07 14:40:53.498475 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.498048 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 07 14:40:53.498475 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.498100 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 07 14:40:53.498475 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.498251 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 07 14:40:53.498821 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.498721 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 07 14:40:53.499095 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.499061 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 07 14:40:53.503053 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.500196 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-in0dqnal7t3k\"" Jul 07 14:40:53.503053 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.500440 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 07 14:40:53.503053 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.500449 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 07 14:40:53.503053 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.500978 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-cdx4k\"" Jul 07 14:40:53.503053 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.501197 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 07 14:40:53.503053 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.503045 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 07 14:40:53.506662 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.506645 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 07 14:40:53.516019 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.513273 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 14:40:53.621423 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.621393 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.621423 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.621424 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.621602 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.621444 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-config-out\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.621602 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.621461 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4j45\" (UniqueName: \"kubernetes.io/projected/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-kube-api-access-h4j45\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.621602 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.621527 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-config\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.621602 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.621554 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.621602 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.621573 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.621602 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.621591 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.621818 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.621614 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.621818 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.621635 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.621818 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.621652 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.621818 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.621676 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.621818 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.621693 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.621818 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.621722 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.621818 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.621747 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.621818 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.621783 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.621818 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.621797 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-web-config\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.621818 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.621812 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.622192 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.621830 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.622192 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.621898 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.722589 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.722531 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4j45\" (UniqueName: \"kubernetes.io/projected/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-kube-api-access-h4j45\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.722589 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.722562 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-config\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.722589 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.722579 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.723164 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.722595 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.723164 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.722612 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.723164 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.722632 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.723164 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.722648 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.723164 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.722932 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.723164 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.722995 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.723164 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.723033 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.723164 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.723072 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.723164 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.723100 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.723164 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.723124 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.723164 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.723146 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-web-config\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.723697 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.723173 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.723697 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.723205 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.723697 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.723243 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.723697 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.723272 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.723697 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.723300 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.723697 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.723310 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.723697 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.723370 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-config-out\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.723697 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.723434 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.724137 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.724112 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.724894 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.724259 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.724894 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.724576 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.724894 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.724769 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.725139 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.725107 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.725915 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.725849 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-config-out\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.727374 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.727184 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.727374 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.727274 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.727374 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.727292 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.727374 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.727327 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-config\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.727374 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.727347 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.728039 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.728019 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.728605 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.728204 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.728605 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.728548 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.728780 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.728674 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.728780 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.728706 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.729193 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.729176 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-web-config\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.731594 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.731577 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4j45\" (UniqueName: \"kubernetes.io/projected/d6cf9f09-a0c7-45a6-9330-a10ae98c2906-kube-api-access-h4j45\") pod \"prometheus-k8s-0\" (UID: \"d6cf9f09-a0c7-45a6-9330-a10ae98c2906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.807299 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.807276 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:40:53.940097 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:53.940076 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 14:40:53.942113 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:40:53.942078 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6cf9f09_a0c7_45a6_9330_a10ae98c2906.slice/crio-3eef0558288ee9c2aec147fc7cd11c093947a1e7db95876a3a14a64a3e37ffd9 WatchSource:0}: Error finding container 3eef0558288ee9c2aec147fc7cd11c093947a1e7db95876a3a14a64a3e37ffd9: Status 404 returned error can't find the container with id 3eef0558288ee9c2aec147fc7cd11c093947a1e7db95876a3a14a64a3e37ffd9 Jul 07 14:40:54.440521 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:54.440488 2555 generic.go:356] "Generic (PLEG): container finished" podID="d6cf9f09-a0c7-45a6-9330-a10ae98c2906" containerID="5907ad94153b1ceec11cf4967b8ab915f9b58eb7401e039d9a650b363b6f22b1" exitCode=0 Jul 07 14:40:54.440688 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:54.440573 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d6cf9f09-a0c7-45a6-9330-a10ae98c2906","Type":"ContainerDied","Data":"5907ad94153b1ceec11cf4967b8ab915f9b58eb7401e039d9a650b363b6f22b1"} Jul 07 14:40:54.440688 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:54.440633 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d6cf9f09-a0c7-45a6-9330-a10ae98c2906","Type":"ContainerStarted","Data":"3eef0558288ee9c2aec147fc7cd11c093947a1e7db95876a3a14a64a3e37ffd9"} Jul 07 14:40:55.041243 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:55.041167 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45780cbd-0927-456b-9745-2cb2e5c8c347" path="/var/lib/kubelet/pods/45780cbd-0927-456b-9745-2cb2e5c8c347/volumes" Jul 07 14:40:55.447578 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:55.447547 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d6cf9f09-a0c7-45a6-9330-a10ae98c2906","Type":"ContainerStarted","Data":"975244fd44cb3b404976c350debe55cfe919f57cf603e3a4ca8df291282e3f0c"} Jul 07 14:40:55.447731 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:55.447609 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d6cf9f09-a0c7-45a6-9330-a10ae98c2906","Type":"ContainerStarted","Data":"1a0acf1a95da5e3add5bc6bcb9745c81ee9e5f1bb033fa34d42f1a1c0b0ac6e8"} Jul 07 14:40:55.447731 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:55.447635 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d6cf9f09-a0c7-45a6-9330-a10ae98c2906","Type":"ContainerStarted","Data":"ab7c192785ab87f720ef481cdc3b82abb017bc456277e81fcd8c24a5fb5dfe02"} Jul 07 14:40:55.447731 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:55.447658 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d6cf9f09-a0c7-45a6-9330-a10ae98c2906","Type":"ContainerStarted","Data":"3175e85f106091c863866170f0bb062e80316bd6dc7344e9ee392fdc9ea511e1"} Jul 07 14:40:55.447731 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:55.447681 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d6cf9f09-a0c7-45a6-9330-a10ae98c2906","Type":"ContainerStarted","Data":"d790dffe267688fd87ce675e74060e7d726fb30adaf974d5bc6384bc98d48abf"} Jul 07 14:40:55.447731 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:55.447707 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d6cf9f09-a0c7-45a6-9330-a10ae98c2906","Type":"ContainerStarted","Data":"aa0756945b0966e4280bb4413fc5a0e3e01faffdc4b7a95f154e0ba63f8ea1fb"} Jul 07 14:40:55.476435 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:55.476375 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.457718227 podStartE2EDuration="2.476356298s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.638071ms" isStatefulPod=false podCreationTimestamp="2026-07-07 14:40:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:40:55.474418194 +0000 UTC m=+396.898604688" watchObservedRunningTime="2026-07-07 14:40:55.476356298 +0000 UTC m=+396.900542774" Jul 07 14:40:58.807609 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:40:58.807574 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:41:53.807808 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:41:53.807704 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:41:53.822869 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:41:53.822846 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:41:54.679595 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:41:54.679567 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:43:46.952247 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:43:46.952160 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8nzr4"] Jul 07 14:43:46.955428 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:43:46.955404 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8nzr4" Jul 07 14:43:46.958112 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:43:46.958080 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 07 14:43:46.959482 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:43:46.959462 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-s8mws\"" Jul 07 14:43:46.959567 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:43:46.959484 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 07 14:43:46.965043 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:43:46.965022 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8nzr4"] Jul 07 14:43:46.999393 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:43:46.999366 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl4qv\" (UniqueName: \"kubernetes.io/projected/9a2430f4-bf79-448a-8e67-0e347e014344-kube-api-access-dl4qv\") pod \"openshift-lws-operator-fd8ccff4c-8nzr4\" (UID: \"9a2430f4-bf79-448a-8e67-0e347e014344\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8nzr4" Jul 07 14:43:46.999493 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:43:46.999421 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a2430f4-bf79-448a-8e67-0e347e014344-tmp\") pod \"openshift-lws-operator-fd8ccff4c-8nzr4\" (UID: \"9a2430f4-bf79-448a-8e67-0e347e014344\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8nzr4" Jul 07 14:43:47.100098 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:43:47.100073 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl4qv\" (UniqueName: \"kubernetes.io/projected/9a2430f4-bf79-448a-8e67-0e347e014344-kube-api-access-dl4qv\") pod \"openshift-lws-operator-fd8ccff4c-8nzr4\" (UID: \"9a2430f4-bf79-448a-8e67-0e347e014344\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8nzr4" Jul 07 14:43:47.100268 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:43:47.100128 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a2430f4-bf79-448a-8e67-0e347e014344-tmp\") pod \"openshift-lws-operator-fd8ccff4c-8nzr4\" (UID: \"9a2430f4-bf79-448a-8e67-0e347e014344\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8nzr4" Jul 07 14:43:47.100483 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:43:47.100465 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a2430f4-bf79-448a-8e67-0e347e014344-tmp\") pod \"openshift-lws-operator-fd8ccff4c-8nzr4\" (UID: \"9a2430f4-bf79-448a-8e67-0e347e014344\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8nzr4" Jul 07 14:43:47.110045 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:43:47.110016 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl4qv\" (UniqueName: \"kubernetes.io/projected/9a2430f4-bf79-448a-8e67-0e347e014344-kube-api-access-dl4qv\") pod \"openshift-lws-operator-fd8ccff4c-8nzr4\" (UID: \"9a2430f4-bf79-448a-8e67-0e347e014344\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8nzr4" Jul 07 14:43:47.264705 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:43:47.264610 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8nzr4" Jul 07 14:43:47.400291 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:43:47.400262 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8nzr4"] Jul 07 14:43:47.402835 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:43:47.402808 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a2430f4_bf79_448a_8e67_0e347e014344.slice/crio-1425813b46c86649ddf869c199011d01af0e6660f3bcc0ca2f27c0d0658b59ee WatchSource:0}: Error finding container 1425813b46c86649ddf869c199011d01af0e6660f3bcc0ca2f27c0d0658b59ee: Status 404 returned error can't find the container with id 1425813b46c86649ddf869c199011d01af0e6660f3bcc0ca2f27c0d0658b59ee Jul 07 14:43:48.098177 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:43:48.098133 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8nzr4" event={"ID":"9a2430f4-bf79-448a-8e67-0e347e014344","Type":"ContainerStarted","Data":"1425813b46c86649ddf869c199011d01af0e6660f3bcc0ca2f27c0d0658b59ee"} Jul 07 14:43:50.108199 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:43:50.108163 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8nzr4" event={"ID":"9a2430f4-bf79-448a-8e67-0e347e014344","Type":"ContainerStarted","Data":"2dda151b820ad5114ce12ab85b50d12259e8171a416aaba709a21c45a323a252"} Jul 07 14:43:50.126457 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:43:50.126118 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8nzr4" podStartSLOduration=1.7818356359999998 podStartE2EDuration="4.126096686s" totalImagesPullingTime="2.34426105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:43:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:43:50.123970321 +0000 UTC m=+571.548156796" watchObservedRunningTime="2026-07-07 14:43:50.126096686 +0000 UTC m=+571.550283163" Jul 07 14:44:09.803031 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:09.802995 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-655bb48ff6-q7dpz"] Jul 07 14:44:09.809208 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:09.809190 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-q7dpz" Jul 07 14:44:09.812965 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:09.812939 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 07 14:44:09.813087 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:09.813060 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 07 14:44:09.813341 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:09.813322 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-g8lnk\"" Jul 07 14:44:09.813519 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:09.813480 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 07 14:44:09.815143 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:09.815123 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 07 14:44:09.838530 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:09.838505 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-655bb48ff6-q7dpz"] Jul 07 14:44:09.902494 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:09.902462 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2620af16-036c-4f11-8e0f-3392581bf237-manifests\") pod \"opendatahub-operator-controller-manager-655bb48ff6-q7dpz\" (UID: \"2620af16-036c-4f11-8e0f-3392581bf237\") " pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-q7dpz" Jul 07 14:44:09.902663 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:09.902510 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2620af16-036c-4f11-8e0f-3392581bf237-webhook-cert\") pod \"opendatahub-operator-controller-manager-655bb48ff6-q7dpz\" (UID: \"2620af16-036c-4f11-8e0f-3392581bf237\") " pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-q7dpz" Jul 07 14:44:09.902663 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:09.902566 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdsv6\" (UniqueName: \"kubernetes.io/projected/2620af16-036c-4f11-8e0f-3392581bf237-kube-api-access-rdsv6\") pod \"opendatahub-operator-controller-manager-655bb48ff6-q7dpz\" (UID: \"2620af16-036c-4f11-8e0f-3392581bf237\") " pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-q7dpz" Jul 07 14:44:09.902663 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:09.902592 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2620af16-036c-4f11-8e0f-3392581bf237-tmp\") pod \"opendatahub-operator-controller-manager-655bb48ff6-q7dpz\" (UID: \"2620af16-036c-4f11-8e0f-3392581bf237\") " pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-q7dpz" Jul 07 14:44:09.902663 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:09.902645 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2620af16-036c-4f11-8e0f-3392581bf237-apiservice-cert\") pod \"opendatahub-operator-controller-manager-655bb48ff6-q7dpz\" (UID: \"2620af16-036c-4f11-8e0f-3392581bf237\") " pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-q7dpz" Jul 07 14:44:10.003037 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:10.003005 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdsv6\" (UniqueName: \"kubernetes.io/projected/2620af16-036c-4f11-8e0f-3392581bf237-kube-api-access-rdsv6\") pod \"opendatahub-operator-controller-manager-655bb48ff6-q7dpz\" (UID: \"2620af16-036c-4f11-8e0f-3392581bf237\") " pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-q7dpz" Jul 07 14:44:10.003037 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:10.003042 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2620af16-036c-4f11-8e0f-3392581bf237-tmp\") pod \"opendatahub-operator-controller-manager-655bb48ff6-q7dpz\" (UID: \"2620af16-036c-4f11-8e0f-3392581bf237\") " pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-q7dpz" Jul 07 14:44:10.003300 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:10.003178 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2620af16-036c-4f11-8e0f-3392581bf237-apiservice-cert\") pod \"opendatahub-operator-controller-manager-655bb48ff6-q7dpz\" (UID: \"2620af16-036c-4f11-8e0f-3392581bf237\") " pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-q7dpz" Jul 07 14:44:10.003300 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:10.003285 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2620af16-036c-4f11-8e0f-3392581bf237-manifests\") pod \"opendatahub-operator-controller-manager-655bb48ff6-q7dpz\" (UID: \"2620af16-036c-4f11-8e0f-3392581bf237\") " pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-q7dpz" Jul 07 14:44:10.003405 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:10.003350 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2620af16-036c-4f11-8e0f-3392581bf237-webhook-cert\") pod \"opendatahub-operator-controller-manager-655bb48ff6-q7dpz\" (UID: \"2620af16-036c-4f11-8e0f-3392581bf237\") " pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-q7dpz" Jul 07 14:44:10.003598 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:10.003577 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2620af16-036c-4f11-8e0f-3392581bf237-manifests\") pod \"opendatahub-operator-controller-manager-655bb48ff6-q7dpz\" (UID: \"2620af16-036c-4f11-8e0f-3392581bf237\") " pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-q7dpz" Jul 07 14:44:10.005299 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:10.005277 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2620af16-036c-4f11-8e0f-3392581bf237-tmp\") pod \"opendatahub-operator-controller-manager-655bb48ff6-q7dpz\" (UID: \"2620af16-036c-4f11-8e0f-3392581bf237\") " pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-q7dpz" Jul 07 14:44:10.005565 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:10.005548 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2620af16-036c-4f11-8e0f-3392581bf237-apiservice-cert\") pod \"opendatahub-operator-controller-manager-655bb48ff6-q7dpz\" (UID: \"2620af16-036c-4f11-8e0f-3392581bf237\") " pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-q7dpz" Jul 07 14:44:10.005622 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:10.005602 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2620af16-036c-4f11-8e0f-3392581bf237-webhook-cert\") pod \"opendatahub-operator-controller-manager-655bb48ff6-q7dpz\" (UID: \"2620af16-036c-4f11-8e0f-3392581bf237\") " pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-q7dpz" Jul 07 14:44:10.012561 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:10.012539 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdsv6\" (UniqueName: \"kubernetes.io/projected/2620af16-036c-4f11-8e0f-3392581bf237-kube-api-access-rdsv6\") pod \"opendatahub-operator-controller-manager-655bb48ff6-q7dpz\" (UID: \"2620af16-036c-4f11-8e0f-3392581bf237\") " pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-q7dpz" Jul 07 14:44:10.120309 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:10.120264 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-q7dpz" Jul 07 14:44:10.257941 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:10.257842 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-655bb48ff6-q7dpz"] Jul 07 14:44:10.260433 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:44:10.260407 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2620af16_036c_4f11_8e0f_3392581bf237.slice/crio-d3239f90192c7814dbee7c19b7e307f181810477cf0f94b9a53511029b90f8cc WatchSource:0}: Error finding container d3239f90192c7814dbee7c19b7e307f181810477cf0f94b9a53511029b90f8cc: Status 404 returned error can't find the container with id d3239f90192c7814dbee7c19b7e307f181810477cf0f94b9a53511029b90f8cc Jul 07 14:44:11.190655 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:11.190614 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-q7dpz" event={"ID":"2620af16-036c-4f11-8e0f-3392581bf237","Type":"ContainerStarted","Data":"d3239f90192c7814dbee7c19b7e307f181810477cf0f94b9a53511029b90f8cc"} Jul 07 14:44:14.206109 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:14.206067 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-q7dpz" event={"ID":"2620af16-036c-4f11-8e0f-3392581bf237","Type":"ContainerStarted","Data":"74c89843f981f6b785321b6dbab51f825a3a874d0385fd690379bc10be9b2814"} Jul 07 14:44:15.211851 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:15.211815 2555 generic.go:356] "Generic (PLEG): container finished" podID="2620af16-036c-4f11-8e0f-3392581bf237" containerID="74c89843f981f6b785321b6dbab51f825a3a874d0385fd690379bc10be9b2814" exitCode=0 Jul 07 14:44:15.212342 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:15.211901 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-q7dpz" event={"ID":"2620af16-036c-4f11-8e0f-3392581bf237","Type":"ContainerDied","Data":"74c89843f981f6b785321b6dbab51f825a3a874d0385fd690379bc10be9b2814"} Jul 07 14:44:16.223925 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:16.223874 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-q7dpz" event={"ID":"2620af16-036c-4f11-8e0f-3392581bf237","Type":"ContainerStarted","Data":"7597cdf4ebddbb04756da7efc975d0675de6900a4f2300bab21c8959928227f9"} Jul 07 14:44:16.224402 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:16.223984 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-q7dpz" Jul 07 14:44:16.257995 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:16.257938 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-q7dpz" podStartSLOduration=2.584370544 podStartE2EDuration="7.257921755s" totalImagesPullingTime="3.589828912s" totalInitContainerRuntime="1.083722299s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:44:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:44:16.255639041 +0000 UTC m=+597.679825517" watchObservedRunningTime="2026-07-07 14:44:16.257921755 +0000 UTC m=+597.682108231" Jul 07 14:44:27.231137 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:27.231107 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-q7dpz" Jul 07 14:44:36.743990 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:36.743954 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vncl5"] Jul 07 14:44:36.747222 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:36.747206 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vncl5" Jul 07 14:44:36.751190 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:36.751168 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 07 14:44:36.751314 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:36.751170 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-84rhl\"" Jul 07 14:44:36.760524 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:36.760504 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vncl5"] Jul 07 14:44:36.807955 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:36.807928 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/645fefe7-23f5-414c-b03e-bbba6e8ce6ca-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vncl5\" (UID: \"645fefe7-23f5-414c-b03e-bbba6e8ce6ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vncl5" Jul 07 14:44:36.808144 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:36.807961 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/645fefe7-23f5-414c-b03e-bbba6e8ce6ca-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vncl5\" (UID: \"645fefe7-23f5-414c-b03e-bbba6e8ce6ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vncl5" Jul 07 14:44:36.808144 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:36.807992 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/645fefe7-23f5-414c-b03e-bbba6e8ce6ca-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vncl5\" (UID: \"645fefe7-23f5-414c-b03e-bbba6e8ce6ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vncl5" Jul 07 14:44:36.808144 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:36.808011 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/645fefe7-23f5-414c-b03e-bbba6e8ce6ca-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vncl5\" (UID: \"645fefe7-23f5-414c-b03e-bbba6e8ce6ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vncl5" Jul 07 14:44:36.808144 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:36.808072 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/645fefe7-23f5-414c-b03e-bbba6e8ce6ca-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vncl5\" (UID: \"645fefe7-23f5-414c-b03e-bbba6e8ce6ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vncl5" Jul 07 14:44:36.808326 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:36.808146 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw9cz\" (UniqueName: \"kubernetes.io/projected/645fefe7-23f5-414c-b03e-bbba6e8ce6ca-kube-api-access-dw9cz\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vncl5\" (UID: \"645fefe7-23f5-414c-b03e-bbba6e8ce6ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vncl5" Jul 07 14:44:36.808326 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:36.808186 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/645fefe7-23f5-414c-b03e-bbba6e8ce6ca-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vncl5\" (UID: \"645fefe7-23f5-414c-b03e-bbba6e8ce6ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vncl5" Jul 07 14:44:36.808326 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:36.808224 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/645fefe7-23f5-414c-b03e-bbba6e8ce6ca-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vncl5\" (UID: \"645fefe7-23f5-414c-b03e-bbba6e8ce6ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vncl5" Jul 07 14:44:36.808326 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:36.808254 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/645fefe7-23f5-414c-b03e-bbba6e8ce6ca-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vncl5\" (UID: \"645fefe7-23f5-414c-b03e-bbba6e8ce6ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vncl5" Jul 07 14:44:36.908653 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:36.908618 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/645fefe7-23f5-414c-b03e-bbba6e8ce6ca-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vncl5\" (UID: \"645fefe7-23f5-414c-b03e-bbba6e8ce6ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vncl5" Jul 07 14:44:36.908879 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:36.908664 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/645fefe7-23f5-414c-b03e-bbba6e8ce6ca-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vncl5\" (UID: \"645fefe7-23f5-414c-b03e-bbba6e8ce6ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vncl5" Jul 07 14:44:36.908879 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:36.908791 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/645fefe7-23f5-414c-b03e-bbba6e8ce6ca-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vncl5\" (UID: \"645fefe7-23f5-414c-b03e-bbba6e8ce6ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vncl5" Jul 07 14:44:36.908879 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:36.908820 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/645fefe7-23f5-414c-b03e-bbba6e8ce6ca-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vncl5\" (UID: \"645fefe7-23f5-414c-b03e-bbba6e8ce6ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vncl5" Jul 07 14:44:36.908879 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:36.908845 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/645fefe7-23f5-414c-b03e-bbba6e8ce6ca-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vncl5\" (UID: \"645fefe7-23f5-414c-b03e-bbba6e8ce6ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vncl5" Jul 07 14:44:36.909110 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:36.908896 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw9cz\" (UniqueName: \"kubernetes.io/projected/645fefe7-23f5-414c-b03e-bbba6e8ce6ca-kube-api-access-dw9cz\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vncl5\" (UID: \"645fefe7-23f5-414c-b03e-bbba6e8ce6ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vncl5" Jul 07 14:44:36.909110 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:36.908928 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/645fefe7-23f5-414c-b03e-bbba6e8ce6ca-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vncl5\" (UID: \"645fefe7-23f5-414c-b03e-bbba6e8ce6ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vncl5" Jul 07 14:44:36.909218 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:36.909116 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/645fefe7-23f5-414c-b03e-bbba6e8ce6ca-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vncl5\" (UID: \"645fefe7-23f5-414c-b03e-bbba6e8ce6ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vncl5" Jul 07 14:44:36.909218 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:36.909175 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/645fefe7-23f5-414c-b03e-bbba6e8ce6ca-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vncl5\" (UID: \"645fefe7-23f5-414c-b03e-bbba6e8ce6ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vncl5" Jul 07 14:44:36.909218 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:36.909191 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/645fefe7-23f5-414c-b03e-bbba6e8ce6ca-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vncl5\" (UID: \"645fefe7-23f5-414c-b03e-bbba6e8ce6ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vncl5" Jul 07 14:44:36.909413 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:36.909222 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/645fefe7-23f5-414c-b03e-bbba6e8ce6ca-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vncl5\" (UID: \"645fefe7-23f5-414c-b03e-bbba6e8ce6ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vncl5" Jul 07 14:44:36.909413 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:36.909279 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/645fefe7-23f5-414c-b03e-bbba6e8ce6ca-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vncl5\" (UID: \"645fefe7-23f5-414c-b03e-bbba6e8ce6ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vncl5" Jul 07 14:44:36.909534 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:36.909513 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/645fefe7-23f5-414c-b03e-bbba6e8ce6ca-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vncl5\" (UID: \"645fefe7-23f5-414c-b03e-bbba6e8ce6ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vncl5" Jul 07 14:44:36.909534 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:36.909518 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/645fefe7-23f5-414c-b03e-bbba6e8ce6ca-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vncl5\" (UID: \"645fefe7-23f5-414c-b03e-bbba6e8ce6ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vncl5" Jul 07 14:44:36.911595 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:36.911571 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/645fefe7-23f5-414c-b03e-bbba6e8ce6ca-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vncl5\" (UID: \"645fefe7-23f5-414c-b03e-bbba6e8ce6ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vncl5" Jul 07 14:44:36.911702 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:36.911651 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/645fefe7-23f5-414c-b03e-bbba6e8ce6ca-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vncl5\" (UID: \"645fefe7-23f5-414c-b03e-bbba6e8ce6ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vncl5" Jul 07 14:44:36.918889 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:36.918864 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw9cz\" (UniqueName: \"kubernetes.io/projected/645fefe7-23f5-414c-b03e-bbba6e8ce6ca-kube-api-access-dw9cz\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vncl5\" (UID: \"645fefe7-23f5-414c-b03e-bbba6e8ce6ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vncl5" Jul 07 14:44:36.919012 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:36.918916 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/645fefe7-23f5-414c-b03e-bbba6e8ce6ca-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vncl5\" (UID: \"645fefe7-23f5-414c-b03e-bbba6e8ce6ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vncl5" Jul 07 14:44:37.057991 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:37.057906 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vncl5" Jul 07 14:44:37.199133 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:37.199090 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vncl5"] Jul 07 14:44:37.203240 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:44:37.203202 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod645fefe7_23f5_414c_b03e_bbba6e8ce6ca.slice/crio-4984193a7d6101a3174357024e49d47fc40e98445eb278c9618382715dffdf2d WatchSource:0}: Error finding container 4984193a7d6101a3174357024e49d47fc40e98445eb278c9618382715dffdf2d: Status 404 returned error can't find the container with id 4984193a7d6101a3174357024e49d47fc40e98445eb278c9618382715dffdf2d Jul 07 14:44:37.205018 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:37.205003 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 14:44:37.316614 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:37.316525 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vncl5" event={"ID":"645fefe7-23f5-414c-b03e-bbba6e8ce6ca","Type":"ContainerStarted","Data":"4984193a7d6101a3174357024e49d47fc40e98445eb278c9618382715dffdf2d"} Jul 07 14:44:39.196399 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.196364 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6d85f8474c-8mn9v"] Jul 07 14:44:39.203233 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.203209 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6d85f8474c-8mn9v" Jul 07 14:44:39.206950 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.206912 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 07 14:44:39.207415 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.207396 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 07 14:44:39.207898 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.207850 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-8hv5q\"" Jul 07 14:44:39.207898 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.207884 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 07 14:44:39.224985 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.224954 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6d85f8474c-8mn9v"] Jul 07 14:44:39.232451 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.232425 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ea222582-1fd6-4e76-a6cb-18a353b998c3-metrics-cert\") pod \"lws-controller-manager-6d85f8474c-8mn9v\" (UID: \"ea222582-1fd6-4e76-a6cb-18a353b998c3\") " pod="openshift-lws-operator/lws-controller-manager-6d85f8474c-8mn9v" Jul 07 14:44:39.232634 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.232464 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7szdj\" (UniqueName: \"kubernetes.io/projected/ea222582-1fd6-4e76-a6cb-18a353b998c3-kube-api-access-7szdj\") pod \"lws-controller-manager-6d85f8474c-8mn9v\" (UID: \"ea222582-1fd6-4e76-a6cb-18a353b998c3\") " pod="openshift-lws-operator/lws-controller-manager-6d85f8474c-8mn9v" Jul 07 14:44:39.232634 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.232496 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ea222582-1fd6-4e76-a6cb-18a353b998c3-manager-config\") pod \"lws-controller-manager-6d85f8474c-8mn9v\" (UID: \"ea222582-1fd6-4e76-a6cb-18a353b998c3\") " pod="openshift-lws-operator/lws-controller-manager-6d85f8474c-8mn9v" Jul 07 14:44:39.232634 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.232575 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea222582-1fd6-4e76-a6cb-18a353b998c3-cert\") pod \"lws-controller-manager-6d85f8474c-8mn9v\" (UID: \"ea222582-1fd6-4e76-a6cb-18a353b998c3\") " pod="openshift-lws-operator/lws-controller-manager-6d85f8474c-8mn9v" Jul 07 14:44:39.333829 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.333791 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ea222582-1fd6-4e76-a6cb-18a353b998c3-metrics-cert\") pod \"lws-controller-manager-6d85f8474c-8mn9v\" (UID: \"ea222582-1fd6-4e76-a6cb-18a353b998c3\") " pod="openshift-lws-operator/lws-controller-manager-6d85f8474c-8mn9v" Jul 07 14:44:39.334025 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.333841 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7szdj\" (UniqueName: \"kubernetes.io/projected/ea222582-1fd6-4e76-a6cb-18a353b998c3-kube-api-access-7szdj\") pod \"lws-controller-manager-6d85f8474c-8mn9v\" (UID: \"ea222582-1fd6-4e76-a6cb-18a353b998c3\") " pod="openshift-lws-operator/lws-controller-manager-6d85f8474c-8mn9v" Jul 07 14:44:39.334025 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.333873 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ea222582-1fd6-4e76-a6cb-18a353b998c3-manager-config\") pod \"lws-controller-manager-6d85f8474c-8mn9v\" (UID: \"ea222582-1fd6-4e76-a6cb-18a353b998c3\") " pod="openshift-lws-operator/lws-controller-manager-6d85f8474c-8mn9v" Jul 07 14:44:39.334025 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.333916 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea222582-1fd6-4e76-a6cb-18a353b998c3-cert\") pod \"lws-controller-manager-6d85f8474c-8mn9v\" (UID: \"ea222582-1fd6-4e76-a6cb-18a353b998c3\") " pod="openshift-lws-operator/lws-controller-manager-6d85f8474c-8mn9v" Jul 07 14:44:39.334941 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.334911 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ea222582-1fd6-4e76-a6cb-18a353b998c3-manager-config\") pod \"lws-controller-manager-6d85f8474c-8mn9v\" (UID: \"ea222582-1fd6-4e76-a6cb-18a353b998c3\") " pod="openshift-lws-operator/lws-controller-manager-6d85f8474c-8mn9v" Jul 07 14:44:39.336611 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.336591 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ea222582-1fd6-4e76-a6cb-18a353b998c3-metrics-cert\") pod \"lws-controller-manager-6d85f8474c-8mn9v\" (UID: \"ea222582-1fd6-4e76-a6cb-18a353b998c3\") " pod="openshift-lws-operator/lws-controller-manager-6d85f8474c-8mn9v" Jul 07 14:44:39.336815 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.336789 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea222582-1fd6-4e76-a6cb-18a353b998c3-cert\") pod \"lws-controller-manager-6d85f8474c-8mn9v\" (UID: \"ea222582-1fd6-4e76-a6cb-18a353b998c3\") " pod="openshift-lws-operator/lws-controller-manager-6d85f8474c-8mn9v" Jul 07 14:44:39.342995 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.342970 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7szdj\" (UniqueName: \"kubernetes.io/projected/ea222582-1fd6-4e76-a6cb-18a353b998c3-kube-api-access-7szdj\") pod \"lws-controller-manager-6d85f8474c-8mn9v\" (UID: \"ea222582-1fd6-4e76-a6cb-18a353b998c3\") " pod="openshift-lws-operator/lws-controller-manager-6d85f8474c-8mn9v" Jul 07 14:44:39.514416 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.514323 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6d85f8474c-8mn9v" Jul 07 14:44:39.644809 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.642062 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-p7bmz"] Jul 07 14:44:39.650490 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.650457 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-p7bmz" Jul 07 14:44:39.655405 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.655375 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 07 14:44:39.655694 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.655668 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 07 14:44:39.655830 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.655701 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 07 14:44:39.658076 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.658048 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-x697n\"" Jul 07 14:44:39.678644 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.678612 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-p7bmz"] Jul 07 14:44:39.736863 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.736819 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 14:44:39.736966 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.736917 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 14:44:39.737012 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.736965 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 14:44:39.737821 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.737794 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/84e8a09b-f81e-409b-87cd-b754240f58f8-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-p7bmz\" (UID: \"84e8a09b-f81e-409b-87cd-b754240f58f8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-p7bmz" Jul 07 14:44:39.737908 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.737841 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/84e8a09b-f81e-409b-87cd-b754240f58f8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-p7bmz\" (UID: \"84e8a09b-f81e-409b-87cd-b754240f58f8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-p7bmz" Jul 07 14:44:39.737908 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.737875 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/84e8a09b-f81e-409b-87cd-b754240f58f8-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-p7bmz\" (UID: \"84e8a09b-f81e-409b-87cd-b754240f58f8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-p7bmz" Jul 07 14:44:39.738004 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.737954 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5xth\" (UniqueName: \"kubernetes.io/projected/84e8a09b-f81e-409b-87cd-b754240f58f8-kube-api-access-j5xth\") pod \"istiod-openshift-gateway-94bb8fbfd-p7bmz\" (UID: \"84e8a09b-f81e-409b-87cd-b754240f58f8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-p7bmz" Jul 07 14:44:39.738004 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.737983 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/84e8a09b-f81e-409b-87cd-b754240f58f8-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-p7bmz\" (UID: \"84e8a09b-f81e-409b-87cd-b754240f58f8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-p7bmz" Jul 07 14:44:39.738106 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.738005 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/84e8a09b-f81e-409b-87cd-b754240f58f8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-p7bmz\" (UID: \"84e8a09b-f81e-409b-87cd-b754240f58f8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-p7bmz" Jul 07 14:44:39.738106 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.738061 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84e8a09b-f81e-409b-87cd-b754240f58f8-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-p7bmz\" (UID: \"84e8a09b-f81e-409b-87cd-b754240f58f8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-p7bmz" Jul 07 14:44:39.814090 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.814061 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6d85f8474c-8mn9v"] Jul 07 14:44:39.816500 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:44:39.816476 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea222582_1fd6_4e76_a6cb_18a353b998c3.slice/crio-4261286e072283f97a4f0abf949b89e5cdd4c70238788bfce61cf9e3ec656286 WatchSource:0}: Error finding container 4261286e072283f97a4f0abf949b89e5cdd4c70238788bfce61cf9e3ec656286: Status 404 returned error can't find the container with id 4261286e072283f97a4f0abf949b89e5cdd4c70238788bfce61cf9e3ec656286 Jul 07 14:44:39.839465 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.839440 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/84e8a09b-f81e-409b-87cd-b754240f58f8-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-p7bmz\" (UID: \"84e8a09b-f81e-409b-87cd-b754240f58f8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-p7bmz" Jul 07 14:44:39.839606 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.839477 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/84e8a09b-f81e-409b-87cd-b754240f58f8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-p7bmz\" (UID: \"84e8a09b-f81e-409b-87cd-b754240f58f8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-p7bmz" Jul 07 14:44:39.839606 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.839508 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/84e8a09b-f81e-409b-87cd-b754240f58f8-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-p7bmz\" (UID: \"84e8a09b-f81e-409b-87cd-b754240f58f8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-p7bmz" Jul 07 14:44:39.839606 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.839553 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5xth\" (UniqueName: \"kubernetes.io/projected/84e8a09b-f81e-409b-87cd-b754240f58f8-kube-api-access-j5xth\") pod \"istiod-openshift-gateway-94bb8fbfd-p7bmz\" (UID: \"84e8a09b-f81e-409b-87cd-b754240f58f8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-p7bmz" Jul 07 14:44:39.839606 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.839573 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/84e8a09b-f81e-409b-87cd-b754240f58f8-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-p7bmz\" (UID: \"84e8a09b-f81e-409b-87cd-b754240f58f8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-p7bmz" Jul 07 14:44:39.839606 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.839589 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/84e8a09b-f81e-409b-87cd-b754240f58f8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-p7bmz\" (UID: \"84e8a09b-f81e-409b-87cd-b754240f58f8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-p7bmz" Jul 07 14:44:39.839898 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.839620 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84e8a09b-f81e-409b-87cd-b754240f58f8-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-p7bmz\" (UID: \"84e8a09b-f81e-409b-87cd-b754240f58f8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-p7bmz" Jul 07 14:44:39.840806 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.840357 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/84e8a09b-f81e-409b-87cd-b754240f58f8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-p7bmz\" (UID: \"84e8a09b-f81e-409b-87cd-b754240f58f8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-p7bmz" Jul 07 14:44:39.842400 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.842373 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/84e8a09b-f81e-409b-87cd-b754240f58f8-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-p7bmz\" (UID: \"84e8a09b-f81e-409b-87cd-b754240f58f8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-p7bmz" Jul 07 14:44:39.842572 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.842554 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/84e8a09b-f81e-409b-87cd-b754240f58f8-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-p7bmz\" (UID: \"84e8a09b-f81e-409b-87cd-b754240f58f8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-p7bmz" Jul 07 14:44:39.842736 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.842714 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/84e8a09b-f81e-409b-87cd-b754240f58f8-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-p7bmz\" (UID: \"84e8a09b-f81e-409b-87cd-b754240f58f8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-p7bmz" Jul 07 14:44:39.843273 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.843253 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/84e8a09b-f81e-409b-87cd-b754240f58f8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-p7bmz\" (UID: \"84e8a09b-f81e-409b-87cd-b754240f58f8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-p7bmz" Jul 07 14:44:39.849048 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.849016 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84e8a09b-f81e-409b-87cd-b754240f58f8-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-p7bmz\" (UID: \"84e8a09b-f81e-409b-87cd-b754240f58f8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-p7bmz" Jul 07 14:44:39.849639 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.849617 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5xth\" (UniqueName: \"kubernetes.io/projected/84e8a09b-f81e-409b-87cd-b754240f58f8-kube-api-access-j5xth\") pod \"istiod-openshift-gateway-94bb8fbfd-p7bmz\" (UID: \"84e8a09b-f81e-409b-87cd-b754240f58f8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-p7bmz" Jul 07 14:44:39.960774 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:39.960722 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-p7bmz" Jul 07 14:44:40.140089 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:40.140063 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-p7bmz"] Jul 07 14:44:40.140833 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:44:40.140808 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84e8a09b_f81e_409b_87cd_b754240f58f8.slice/crio-656fb91fce324a15f6ab095794c44ef3bc7c52245c9a62468dfbb23efdeb2297 WatchSource:0}: Error finding container 656fb91fce324a15f6ab095794c44ef3bc7c52245c9a62468dfbb23efdeb2297: Status 404 returned error can't find the container with id 656fb91fce324a15f6ab095794c44ef3bc7c52245c9a62468dfbb23efdeb2297 Jul 07 14:44:40.330962 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:40.330928 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-p7bmz" event={"ID":"84e8a09b-f81e-409b-87cd-b754240f58f8","Type":"ContainerStarted","Data":"656fb91fce324a15f6ab095794c44ef3bc7c52245c9a62468dfbb23efdeb2297"} Jul 07 14:44:40.332005 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:40.331972 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6d85f8474c-8mn9v" event={"ID":"ea222582-1fd6-4e76-a6cb-18a353b998c3","Type":"ContainerStarted","Data":"4261286e072283f97a4f0abf949b89e5cdd4c70238788bfce61cf9e3ec656286"} Jul 07 14:44:40.333358 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:40.333334 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vncl5" event={"ID":"645fefe7-23f5-414c-b03e-bbba6e8ce6ca","Type":"ContainerStarted","Data":"f359cb13f204a78f77b819a791257aa84e23b1ddfc130a5a89fad9cb462f3a8d"} Jul 07 14:44:40.362716 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:40.362658 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vncl5" podStartSLOduration=1.831304468 podStartE2EDuration="4.362641521s" totalImagesPullingTime="2.531337053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:44:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:44:40.360728667 +0000 UTC m=+621.784915143" watchObservedRunningTime="2026-07-07 14:44:40.362641521 +0000 UTC m=+621.786827995" Jul 07 14:44:41.058407 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:41.058323 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vncl5" Jul 07 14:44:41.064265 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:41.064207 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vncl5" Jul 07 14:44:41.338260 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:41.338231 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vncl5" Jul 07 14:44:41.339469 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:41.339436 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vncl5" Jul 07 14:44:42.345144 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:42.345106 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6d85f8474c-8mn9v" event={"ID":"ea222582-1fd6-4e76-a6cb-18a353b998c3","Type":"ContainerStarted","Data":"7bec85932d1b58707646d60389cec97940080eee58fc0871f4ecc8a052131f6f"} Jul 07 14:44:42.345695 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:42.345172 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6d85f8474c-8mn9v" Jul 07 14:44:42.367560 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:42.367503 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6d85f8474c-8mn9v" podStartSLOduration=1.6955099310000001 podStartE2EDuration="3.367480191s" totalImagesPullingTime="1.67197026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:44:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:44:42.36558542 +0000 UTC m=+623.789771896" watchObservedRunningTime="2026-07-07 14:44:42.367480191 +0000 UTC m=+623.791666667" Jul 07 14:44:43.524589 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:43.524560 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 14:44:43.524861 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:43.524623 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 14:44:44.355301 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:44.355260 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-p7bmz" event={"ID":"84e8a09b-f81e-409b-87cd-b754240f58f8","Type":"ContainerStarted","Data":"221e65cd4415ad612ea2f2bd59846a0eee3cb5688e78f9e5308ab6053b54f3e8"} Jul 07 14:44:44.355498 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:44.355459 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-p7bmz" Jul 07 14:44:44.357182 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:44.357155 2555 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-p7bmz container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 07 14:44:44.357319 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:44.357207 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-p7bmz" podUID="84e8a09b-f81e-409b-87cd-b754240f58f8" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 14:44:44.386386 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:44.386334 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-p7bmz" podStartSLOduration=2.004822037 podStartE2EDuration="5.38631946s" totalImagesPullingTime="3.381497423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:44:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:44:44.385216292 +0000 UTC m=+625.809402789" watchObservedRunningTime="2026-07-07 14:44:44.38631946 +0000 UTC m=+625.810505934" Jul 07 14:44:45.360480 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:45.360444 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-p7bmz" Jul 07 14:44:53.352582 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:44:53.352548 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6d85f8474c-8mn9v" Jul 07 14:48:05.289189 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:05.289152 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-58bd56c68f-cpgph"] Jul 07 14:48:05.292601 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:05.292582 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58bd56c68f-cpgph" Jul 07 14:48:05.295320 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:05.295302 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 07 14:48:05.295437 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:05.295376 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mhvhq\"" Jul 07 14:48:05.295532 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:05.295520 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 07 14:48:05.306729 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:05.306706 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58bd56c68f-cpgph"] Jul 07 14:48:05.445434 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:05.445398 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzrlb\" (UniqueName: \"kubernetes.io/projected/a52c6ae6-24d7-48e4-8cec-061753487fd0-kube-api-access-dzrlb\") pod \"maas-controller-58bd56c68f-cpgph\" (UID: \"a52c6ae6-24d7-48e4-8cec-061753487fd0\") " pod="opendatahub/maas-controller-58bd56c68f-cpgph" Jul 07 14:48:05.445607 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:05.445439 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a52c6ae6-24d7-48e4-8cec-061753487fd0-webhook-cert\") pod \"maas-controller-58bd56c68f-cpgph\" (UID: \"a52c6ae6-24d7-48e4-8cec-061753487fd0\") " pod="opendatahub/maas-controller-58bd56c68f-cpgph" Jul 07 14:48:05.527008 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:05.526973 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5b667b448b-6n6r7"] Jul 07 14:48:05.530393 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:05.530378 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b667b448b-6n6r7" Jul 07 14:48:05.547411 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:05.547343 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzrlb\" (UniqueName: \"kubernetes.io/projected/a52c6ae6-24d7-48e4-8cec-061753487fd0-kube-api-access-dzrlb\") pod \"maas-controller-58bd56c68f-cpgph\" (UID: \"a52c6ae6-24d7-48e4-8cec-061753487fd0\") " pod="opendatahub/maas-controller-58bd56c68f-cpgph" Jul 07 14:48:05.547411 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:05.547405 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a52c6ae6-24d7-48e4-8cec-061753487fd0-webhook-cert\") pod \"maas-controller-58bd56c68f-cpgph\" (UID: \"a52c6ae6-24d7-48e4-8cec-061753487fd0\") " pod="opendatahub/maas-controller-58bd56c68f-cpgph" Jul 07 14:48:05.550679 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:05.550615 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a52c6ae6-24d7-48e4-8cec-061753487fd0-webhook-cert\") pod \"maas-controller-58bd56c68f-cpgph\" (UID: \"a52c6ae6-24d7-48e4-8cec-061753487fd0\") " pod="opendatahub/maas-controller-58bd56c68f-cpgph" Jul 07 14:48:05.555698 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:05.555673 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b667b448b-6n6r7"] Jul 07 14:48:05.577027 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:05.577001 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzrlb\" (UniqueName: \"kubernetes.io/projected/a52c6ae6-24d7-48e4-8cec-061753487fd0-kube-api-access-dzrlb\") pod \"maas-controller-58bd56c68f-cpgph\" (UID: \"a52c6ae6-24d7-48e4-8cec-061753487fd0\") " pod="opendatahub/maas-controller-58bd56c68f-cpgph" Jul 07 14:48:05.603682 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:05.603650 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58bd56c68f-cpgph" Jul 07 14:48:05.648354 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:05.648316 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/85a6e8e4-4b2e-46a0-8589-3436e1b759e1-webhook-cert\") pod \"maas-controller-5b667b448b-6n6r7\" (UID: \"85a6e8e4-4b2e-46a0-8589-3436e1b759e1\") " pod="opendatahub/maas-controller-5b667b448b-6n6r7" Jul 07 14:48:05.648501 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:05.648429 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhzz2\" (UniqueName: \"kubernetes.io/projected/85a6e8e4-4b2e-46a0-8589-3436e1b759e1-kube-api-access-rhzz2\") pod \"maas-controller-5b667b448b-6n6r7\" (UID: \"85a6e8e4-4b2e-46a0-8589-3436e1b759e1\") " pod="opendatahub/maas-controller-5b667b448b-6n6r7" Jul 07 14:48:05.736923 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:05.736859 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58bd56c68f-cpgph"] Jul 07 14:48:05.739701 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:48:05.739666 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda52c6ae6_24d7_48e4_8cec_061753487fd0.slice/crio-00fffa397924c80cb743d98583b7021955b4aa78227e74d9675dc75eaaabc5dd WatchSource:0}: Error finding container 00fffa397924c80cb743d98583b7021955b4aa78227e74d9675dc75eaaabc5dd: Status 404 returned error can't find the container with id 00fffa397924c80cb743d98583b7021955b4aa78227e74d9675dc75eaaabc5dd Jul 07 14:48:05.749833 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:05.749803 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/85a6e8e4-4b2e-46a0-8589-3436e1b759e1-webhook-cert\") pod \"maas-controller-5b667b448b-6n6r7\" (UID: \"85a6e8e4-4b2e-46a0-8589-3436e1b759e1\") " pod="opendatahub/maas-controller-5b667b448b-6n6r7" Jul 07 14:48:05.749943 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:05.749864 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhzz2\" (UniqueName: \"kubernetes.io/projected/85a6e8e4-4b2e-46a0-8589-3436e1b759e1-kube-api-access-rhzz2\") pod \"maas-controller-5b667b448b-6n6r7\" (UID: \"85a6e8e4-4b2e-46a0-8589-3436e1b759e1\") " pod="opendatahub/maas-controller-5b667b448b-6n6r7" Jul 07 14:48:05.752148 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:05.752123 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/85a6e8e4-4b2e-46a0-8589-3436e1b759e1-webhook-cert\") pod \"maas-controller-5b667b448b-6n6r7\" (UID: \"85a6e8e4-4b2e-46a0-8589-3436e1b759e1\") " pod="opendatahub/maas-controller-5b667b448b-6n6r7" Jul 07 14:48:05.760997 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:05.760975 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhzz2\" (UniqueName: \"kubernetes.io/projected/85a6e8e4-4b2e-46a0-8589-3436e1b759e1-kube-api-access-rhzz2\") pod \"maas-controller-5b667b448b-6n6r7\" (UID: \"85a6e8e4-4b2e-46a0-8589-3436e1b759e1\") " pod="opendatahub/maas-controller-5b667b448b-6n6r7" Jul 07 14:48:05.840030 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:05.840001 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b667b448b-6n6r7" Jul 07 14:48:05.972545 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:05.972517 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b667b448b-6n6r7"] Jul 07 14:48:05.974309 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:48:05.974283 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85a6e8e4_4b2e_46a0_8589_3436e1b759e1.slice/crio-55a46c4f41c4bf68271790d2c278f6b95e8deac5833f0c49c4611a8824238a24 WatchSource:0}: Error finding container 55a46c4f41c4bf68271790d2c278f6b95e8deac5833f0c49c4611a8824238a24: Status 404 returned error can't find the container with id 55a46c4f41c4bf68271790d2c278f6b95e8deac5833f0c49c4611a8824238a24 Jul 07 14:48:06.226148 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:06.226056 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b667b448b-6n6r7" event={"ID":"85a6e8e4-4b2e-46a0-8589-3436e1b759e1","Type":"ContainerStarted","Data":"55a46c4f41c4bf68271790d2c278f6b95e8deac5833f0c49c4611a8824238a24"} Jul 07 14:48:06.229866 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:06.229838 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58bd56c68f-cpgph" event={"ID":"a52c6ae6-24d7-48e4-8cec-061753487fd0","Type":"ContainerStarted","Data":"00fffa397924c80cb743d98583b7021955b4aa78227e74d9675dc75eaaabc5dd"} Jul 07 14:48:09.248806 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:09.248763 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b667b448b-6n6r7" event={"ID":"85a6e8e4-4b2e-46a0-8589-3436e1b759e1","Type":"ContainerStarted","Data":"21bc79d256ebdf7bde6d00cbd556f8895667d24168b39b3dfad6265f65cc6cc6"} Jul 07 14:48:09.249271 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:09.248925 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5b667b448b-6n6r7" Jul 07 14:48:09.250341 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:09.250317 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58bd56c68f-cpgph" event={"ID":"a52c6ae6-24d7-48e4-8cec-061753487fd0","Type":"ContainerStarted","Data":"3f0e40e9b0baa10c0faf854a3b607b0f863115c0a01c6b8cf39661a8561070d2"} Jul 07 14:48:09.250456 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:09.250439 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-58bd56c68f-cpgph" Jul 07 14:48:09.284692 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:09.284640 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5b667b448b-6n6r7" podStartSLOduration=1.665224586 podStartE2EDuration="4.2846249s" totalImagesPullingTime="2.619400314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:48:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:48:09.282435887 +0000 UTC m=+830.706622388" watchObservedRunningTime="2026-07-07 14:48:09.2846249 +0000 UTC m=+830.708811374" Jul 07 14:48:09.313152 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:09.313099 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-58bd56c68f-cpgph" podStartSLOduration=1.469196449 podStartE2EDuration="4.313085523s" totalImagesPullingTime="2.843889074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:48:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:48:09.311122085 +0000 UTC m=+830.735308560" watchObservedRunningTime="2026-07-07 14:48:09.313085523 +0000 UTC m=+830.737271998" Jul 07 14:48:20.264220 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:20.264188 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-58bd56c68f-cpgph" Jul 07 14:48:20.264668 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:20.264262 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5b667b448b-6n6r7" Jul 07 14:48:20.331629 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:20.331595 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58bd56c68f-cpgph"] Jul 07 14:48:20.331827 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:20.331793 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-58bd56c68f-cpgph" podUID="a52c6ae6-24d7-48e4-8cec-061753487fd0" containerName="manager" containerID="cri-o://3f0e40e9b0baa10c0faf854a3b607b0f863115c0a01c6b8cf39661a8561070d2" gracePeriod=30 Jul 07 14:48:20.570559 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:20.570536 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58bd56c68f-cpgph" Jul 07 14:48:20.689577 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:20.689543 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a52c6ae6-24d7-48e4-8cec-061753487fd0-webhook-cert\") pod \"a52c6ae6-24d7-48e4-8cec-061753487fd0\" (UID: \"a52c6ae6-24d7-48e4-8cec-061753487fd0\") " Jul 07 14:48:20.689745 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:20.689610 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzrlb\" (UniqueName: \"kubernetes.io/projected/a52c6ae6-24d7-48e4-8cec-061753487fd0-kube-api-access-dzrlb\") pod \"a52c6ae6-24d7-48e4-8cec-061753487fd0\" (UID: \"a52c6ae6-24d7-48e4-8cec-061753487fd0\") " Jul 07 14:48:20.691770 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:20.691727 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a52c6ae6-24d7-48e4-8cec-061753487fd0-kube-api-access-dzrlb" (OuterVolumeSpecName: "kube-api-access-dzrlb") pod "a52c6ae6-24d7-48e4-8cec-061753487fd0" (UID: "a52c6ae6-24d7-48e4-8cec-061753487fd0"). InnerVolumeSpecName "kube-api-access-dzrlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:48:20.691890 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:20.691776 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a52c6ae6-24d7-48e4-8cec-061753487fd0-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "a52c6ae6-24d7-48e4-8cec-061753487fd0" (UID: "a52c6ae6-24d7-48e4-8cec-061753487fd0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:48:20.790588 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:20.790557 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzrlb\" (UniqueName: \"kubernetes.io/projected/a52c6ae6-24d7-48e4-8cec-061753487fd0-kube-api-access-dzrlb\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:48:20.790588 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:20.790584 2555 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a52c6ae6-24d7-48e4-8cec-061753487fd0-webhook-cert\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:48:21.305373 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:21.305284 2555 generic.go:356] "Generic (PLEG): container finished" podID="a52c6ae6-24d7-48e4-8cec-061753487fd0" containerID="3f0e40e9b0baa10c0faf854a3b607b0f863115c0a01c6b8cf39661a8561070d2" exitCode=0 Jul 07 14:48:21.305373 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:21.305347 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58bd56c68f-cpgph" Jul 07 14:48:21.305898 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:21.305367 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58bd56c68f-cpgph" event={"ID":"a52c6ae6-24d7-48e4-8cec-061753487fd0","Type":"ContainerDied","Data":"3f0e40e9b0baa10c0faf854a3b607b0f863115c0a01c6b8cf39661a8561070d2"} Jul 07 14:48:21.305898 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:21.305422 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58bd56c68f-cpgph" event={"ID":"a52c6ae6-24d7-48e4-8cec-061753487fd0","Type":"ContainerDied","Data":"00fffa397924c80cb743d98583b7021955b4aa78227e74d9675dc75eaaabc5dd"} Jul 07 14:48:21.305898 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:21.305443 2555 scope.go:117] "RemoveContainer" containerID="3f0e40e9b0baa10c0faf854a3b607b0f863115c0a01c6b8cf39661a8561070d2" Jul 07 14:48:21.313780 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:21.313744 2555 scope.go:117] "RemoveContainer" containerID="3f0e40e9b0baa10c0faf854a3b607b0f863115c0a01c6b8cf39661a8561070d2" Jul 07 14:48:21.314070 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:48:21.314052 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f0e40e9b0baa10c0faf854a3b607b0f863115c0a01c6b8cf39661a8561070d2\": container with ID starting with 3f0e40e9b0baa10c0faf854a3b607b0f863115c0a01c6b8cf39661a8561070d2 not found: ID does not exist" containerID="3f0e40e9b0baa10c0faf854a3b607b0f863115c0a01c6b8cf39661a8561070d2" Jul 07 14:48:21.314145 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:21.314078 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f0e40e9b0baa10c0faf854a3b607b0f863115c0a01c6b8cf39661a8561070d2"} err="failed to get container status \"3f0e40e9b0baa10c0faf854a3b607b0f863115c0a01c6b8cf39661a8561070d2\": rpc error: code = NotFound desc = could not find container \"3f0e40e9b0baa10c0faf854a3b607b0f863115c0a01c6b8cf39661a8561070d2\": container with ID starting with 3f0e40e9b0baa10c0faf854a3b607b0f863115c0a01c6b8cf39661a8561070d2 not found: ID does not exist" Jul 07 14:48:21.323163 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:21.323139 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58bd56c68f-cpgph"] Jul 07 14:48:21.327124 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:21.327104 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-58bd56c68f-cpgph"] Jul 07 14:48:23.041540 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:23.041494 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a52c6ae6-24d7-48e4-8cec-061753487fd0" path="/var/lib/kubelet/pods/a52c6ae6-24d7-48e4-8cec-061753487fd0/volumes" Jul 07 14:48:39.810772 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:39.809497 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-767c6f79c5-drmrr"] Jul 07 14:48:39.810772 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:39.810323 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a52c6ae6-24d7-48e4-8cec-061753487fd0" containerName="manager" Jul 07 14:48:39.810772 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:39.810345 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a52c6ae6-24d7-48e4-8cec-061753487fd0" containerName="manager" Jul 07 14:48:39.810772 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:39.810507 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a52c6ae6-24d7-48e4-8cec-061753487fd0" containerName="manager" Jul 07 14:48:39.816035 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:39.816006 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-767c6f79c5-drmrr" Jul 07 14:48:39.819048 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:39.819027 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-gjhm4\"" Jul 07 14:48:39.819165 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:39.819030 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\"" Jul 07 14:48:39.824682 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:39.824655 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-767c6f79c5-drmrr"] Jul 07 14:48:39.954722 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:39.954687 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/776b8309-f958-440a-b03f-3f054c8b8e68-maas-api-tls\") pod \"maas-api-767c6f79c5-drmrr\" (UID: \"776b8309-f958-440a-b03f-3f054c8b8e68\") " pod="opendatahub/maas-api-767c6f79c5-drmrr" Jul 07 14:48:39.954722 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:39.954726 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/776b8309-f958-440a-b03f-3f054c8b8e68-openshift-service-ca\") pod \"maas-api-767c6f79c5-drmrr\" (UID: \"776b8309-f958-440a-b03f-3f054c8b8e68\") " pod="opendatahub/maas-api-767c6f79c5-drmrr" Jul 07 14:48:39.954941 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:39.954833 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2782x\" (UniqueName: \"kubernetes.io/projected/776b8309-f958-440a-b03f-3f054c8b8e68-kube-api-access-2782x\") pod \"maas-api-767c6f79c5-drmrr\" (UID: \"776b8309-f958-440a-b03f-3f054c8b8e68\") " pod="opendatahub/maas-api-767c6f79c5-drmrr" Jul 07 14:48:40.056200 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:40.056169 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/776b8309-f958-440a-b03f-3f054c8b8e68-maas-api-tls\") pod \"maas-api-767c6f79c5-drmrr\" (UID: \"776b8309-f958-440a-b03f-3f054c8b8e68\") " pod="opendatahub/maas-api-767c6f79c5-drmrr" Jul 07 14:48:40.056200 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:40.056203 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/776b8309-f958-440a-b03f-3f054c8b8e68-openshift-service-ca\") pod \"maas-api-767c6f79c5-drmrr\" (UID: \"776b8309-f958-440a-b03f-3f054c8b8e68\") " pod="opendatahub/maas-api-767c6f79c5-drmrr" Jul 07 14:48:40.056397 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:40.056339 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2782x\" (UniqueName: \"kubernetes.io/projected/776b8309-f958-440a-b03f-3f054c8b8e68-kube-api-access-2782x\") pod \"maas-api-767c6f79c5-drmrr\" (UID: \"776b8309-f958-440a-b03f-3f054c8b8e68\") " pod="opendatahub/maas-api-767c6f79c5-drmrr" Jul 07 14:48:40.056800 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:40.056778 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/776b8309-f958-440a-b03f-3f054c8b8e68-openshift-service-ca\") pod \"maas-api-767c6f79c5-drmrr\" (UID: \"776b8309-f958-440a-b03f-3f054c8b8e68\") " pod="opendatahub/maas-api-767c6f79c5-drmrr" Jul 07 14:48:40.058594 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:40.058567 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/776b8309-f958-440a-b03f-3f054c8b8e68-maas-api-tls\") pod \"maas-api-767c6f79c5-drmrr\" (UID: \"776b8309-f958-440a-b03f-3f054c8b8e68\") " pod="opendatahub/maas-api-767c6f79c5-drmrr" Jul 07 14:48:40.067505 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:40.067453 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2782x\" (UniqueName: \"kubernetes.io/projected/776b8309-f958-440a-b03f-3f054c8b8e68-kube-api-access-2782x\") pod \"maas-api-767c6f79c5-drmrr\" (UID: \"776b8309-f958-440a-b03f-3f054c8b8e68\") " pod="opendatahub/maas-api-767c6f79c5-drmrr" Jul 07 14:48:40.127250 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:40.127223 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-767c6f79c5-drmrr" Jul 07 14:48:40.270718 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:40.270691 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-767c6f79c5-drmrr"] Jul 07 14:48:40.272283 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:48:40.272253 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod776b8309_f958_440a_b03f_3f054c8b8e68.slice/crio-6ca329c1a5c43ce1828db986d438d31a46760130e942114256475e182967996b WatchSource:0}: Error finding container 6ca329c1a5c43ce1828db986d438d31a46760130e942114256475e182967996b: Status 404 returned error can't find the container with id 6ca329c1a5c43ce1828db986d438d31a46760130e942114256475e182967996b Jul 07 14:48:40.388835 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:40.388796 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-767c6f79c5-drmrr" event={"ID":"776b8309-f958-440a-b03f-3f054c8b8e68","Type":"ContainerStarted","Data":"6ca329c1a5c43ce1828db986d438d31a46760130e942114256475e182967996b"} Jul 07 14:48:42.398728 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:42.398693 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-767c6f79c5-drmrr" event={"ID":"776b8309-f958-440a-b03f-3f054c8b8e68","Type":"ContainerStarted","Data":"cd2ebc01ab651cd61c87ab83c291db7248da0436380fe15e2e2f371c207558b9"} Jul 07 14:48:42.399104 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:42.398782 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-767c6f79c5-drmrr" Jul 07 14:48:42.417992 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:42.417947 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-767c6f79c5-drmrr" podStartSLOduration=1.892688806 podStartE2EDuration="3.417931542s" totalImagesPullingTime="1.525242736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:48:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:48:42.416460344 +0000 UTC m=+863.840646820" watchObservedRunningTime="2026-07-07 14:48:42.417931542 +0000 UTC m=+863.842118018" Jul 07 14:48:48.408974 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:48.408943 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-767c6f79c5-drmrr" Jul 07 14:48:53.970796 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:53.970744 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw"] Jul 07 14:48:53.974173 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:53.974157 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw" Jul 07 14:48:53.977101 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:53.977072 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 07 14:48:53.978085 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:53.978068 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 07 14:48:53.978174 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:53.978069 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 07 14:48:53.978174 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:53.978097 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-qh5dn\"" Jul 07 14:48:53.985928 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:53.985903 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw"] Jul 07 14:48:54.078808 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:54.078775 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b682a975-ce9b-488a-a3b8-5d33d73cb1c1-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw\" (UID: \"b682a975-ce9b-488a-a3b8-5d33d73cb1c1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw" Jul 07 14:48:54.078808 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:54.078809 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwwjh\" (UniqueName: \"kubernetes.io/projected/b682a975-ce9b-488a-a3b8-5d33d73cb1c1-kube-api-access-rwwjh\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw\" (UID: \"b682a975-ce9b-488a-a3b8-5d33d73cb1c1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw" Jul 07 14:48:54.078999 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:54.078866 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b682a975-ce9b-488a-a3b8-5d33d73cb1c1-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw\" (UID: \"b682a975-ce9b-488a-a3b8-5d33d73cb1c1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw" Jul 07 14:48:54.078999 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:54.078893 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b682a975-ce9b-488a-a3b8-5d33d73cb1c1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw\" (UID: \"b682a975-ce9b-488a-a3b8-5d33d73cb1c1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw" Jul 07 14:48:54.078999 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:54.078966 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b682a975-ce9b-488a-a3b8-5d33d73cb1c1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw\" (UID: \"b682a975-ce9b-488a-a3b8-5d33d73cb1c1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw" Jul 07 14:48:54.079097 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:54.079021 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b682a975-ce9b-488a-a3b8-5d33d73cb1c1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw\" (UID: \"b682a975-ce9b-488a-a3b8-5d33d73cb1c1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw" Jul 07 14:48:54.180099 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:54.180070 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b682a975-ce9b-488a-a3b8-5d33d73cb1c1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw\" (UID: \"b682a975-ce9b-488a-a3b8-5d33d73cb1c1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw" Jul 07 14:48:54.180265 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:54.180131 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b682a975-ce9b-488a-a3b8-5d33d73cb1c1-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw\" (UID: \"b682a975-ce9b-488a-a3b8-5d33d73cb1c1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw" Jul 07 14:48:54.180265 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:54.180159 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwwjh\" (UniqueName: \"kubernetes.io/projected/b682a975-ce9b-488a-a3b8-5d33d73cb1c1-kube-api-access-rwwjh\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw\" (UID: \"b682a975-ce9b-488a-a3b8-5d33d73cb1c1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw" Jul 07 14:48:54.180265 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:54.180183 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b682a975-ce9b-488a-a3b8-5d33d73cb1c1-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw\" (UID: \"b682a975-ce9b-488a-a3b8-5d33d73cb1c1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw" Jul 07 14:48:54.180265 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:54.180204 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b682a975-ce9b-488a-a3b8-5d33d73cb1c1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw\" (UID: \"b682a975-ce9b-488a-a3b8-5d33d73cb1c1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw" Jul 07 14:48:54.180265 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:54.180251 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b682a975-ce9b-488a-a3b8-5d33d73cb1c1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw\" (UID: \"b682a975-ce9b-488a-a3b8-5d33d73cb1c1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw" Jul 07 14:48:54.180559 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:54.180536 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b682a975-ce9b-488a-a3b8-5d33d73cb1c1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw\" (UID: \"b682a975-ce9b-488a-a3b8-5d33d73cb1c1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw" Jul 07 14:48:54.180635 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:54.180604 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b682a975-ce9b-488a-a3b8-5d33d73cb1c1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw\" (UID: \"b682a975-ce9b-488a-a3b8-5d33d73cb1c1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw" Jul 07 14:48:54.180699 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:54.180682 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b682a975-ce9b-488a-a3b8-5d33d73cb1c1-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw\" (UID: \"b682a975-ce9b-488a-a3b8-5d33d73cb1c1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw" Jul 07 14:48:54.182510 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:54.182488 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b682a975-ce9b-488a-a3b8-5d33d73cb1c1-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw\" (UID: \"b682a975-ce9b-488a-a3b8-5d33d73cb1c1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw" Jul 07 14:48:54.182729 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:54.182710 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b682a975-ce9b-488a-a3b8-5d33d73cb1c1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw\" (UID: \"b682a975-ce9b-488a-a3b8-5d33d73cb1c1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw" Jul 07 14:48:54.191445 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:54.191420 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwwjh\" (UniqueName: \"kubernetes.io/projected/b682a975-ce9b-488a-a3b8-5d33d73cb1c1-kube-api-access-rwwjh\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw\" (UID: \"b682a975-ce9b-488a-a3b8-5d33d73cb1c1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw" Jul 07 14:48:54.284490 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:54.284408 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw" Jul 07 14:48:54.416150 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:54.416096 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw"] Jul 07 14:48:54.419976 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:48:54.419944 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb682a975_ce9b_488a_a3b8_5d33d73cb1c1.slice/crio-335e3c6c1eeed527bd3bd01f61b9a985431e965e8b4898d7ed2e76d0effb843d WatchSource:0}: Error finding container 335e3c6c1eeed527bd3bd01f61b9a985431e965e8b4898d7ed2e76d0effb843d: Status 404 returned error can't find the container with id 335e3c6c1eeed527bd3bd01f61b9a985431e965e8b4898d7ed2e76d0effb843d Jul 07 14:48:54.453280 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:54.453249 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw" event={"ID":"b682a975-ce9b-488a-a3b8-5d33d73cb1c1","Type":"ContainerStarted","Data":"335e3c6c1eeed527bd3bd01f61b9a985431e965e8b4898d7ed2e76d0effb843d"} Jul 07 14:48:56.463459 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:56.463421 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw" event={"ID":"b682a975-ce9b-488a-a3b8-5d33d73cb1c1","Type":"ContainerStarted","Data":"179a050327f726481c4db10cbc2224feedf5554d1799b05cbb34e4a003ebcd97"} Jul 07 14:48:56.483603 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:48:56.483532 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw" podStartSLOduration=2.311938801 podStartE2EDuration="3.4835126s" totalImagesPullingTime="1.171573799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:48:56.482936351 +0000 UTC m=+877.907122830" watchObservedRunningTime="2026-07-07 14:48:56.4835126 +0000 UTC m=+877.907699074" Jul 07 14:49:04.285590 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:49:04.285551 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw" Jul 07 14:49:04.286037 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:49:04.285620 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw" Jul 07 14:49:04.297985 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:49:04.297963 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw" Jul 07 14:49:04.510884 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:49:04.510855 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw" Jul 07 14:49:45.257418 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:49:45.257384 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-zwfl5"] Jul 07 14:49:45.260611 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:49:45.260594 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zwfl5" Jul 07 14:49:45.263288 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:49:45.263265 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 07 14:49:45.270677 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:49:45.270653 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-zwfl5"] Jul 07 14:49:45.330849 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:49:45.330821 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c22f9aa-b3b7-4db2-babb-0f9924d3a2c2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zwfl5\" (UID: \"0c22f9aa-b3b7-4db2-babb-0f9924d3a2c2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zwfl5" Jul 07 14:49:45.331018 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:49:45.330870 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c22f9aa-b3b7-4db2-babb-0f9924d3a2c2-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zwfl5\" (UID: \"0c22f9aa-b3b7-4db2-babb-0f9924d3a2c2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zwfl5" Jul 07 14:49:45.331018 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:49:45.330980 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c22f9aa-b3b7-4db2-babb-0f9924d3a2c2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zwfl5\" (UID: \"0c22f9aa-b3b7-4db2-babb-0f9924d3a2c2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zwfl5" Jul 07 14:49:45.331123 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:49:45.331021 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m4mj\" (UniqueName: \"kubernetes.io/projected/0c22f9aa-b3b7-4db2-babb-0f9924d3a2c2-kube-api-access-7m4mj\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zwfl5\" (UID: \"0c22f9aa-b3b7-4db2-babb-0f9924d3a2c2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zwfl5" Jul 07 14:49:45.331123 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:49:45.331077 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c22f9aa-b3b7-4db2-babb-0f9924d3a2c2-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zwfl5\" (UID: \"0c22f9aa-b3b7-4db2-babb-0f9924d3a2c2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zwfl5" Jul 07 14:49:45.331123 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:49:45.331120 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c22f9aa-b3b7-4db2-babb-0f9924d3a2c2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zwfl5\" (UID: \"0c22f9aa-b3b7-4db2-babb-0f9924d3a2c2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zwfl5" Jul 07 14:49:45.432051 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:49:45.432016 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c22f9aa-b3b7-4db2-babb-0f9924d3a2c2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zwfl5\" (UID: \"0c22f9aa-b3b7-4db2-babb-0f9924d3a2c2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zwfl5" Jul 07 14:49:45.432219 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:49:45.432076 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m4mj\" (UniqueName: \"kubernetes.io/projected/0c22f9aa-b3b7-4db2-babb-0f9924d3a2c2-kube-api-access-7m4mj\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zwfl5\" (UID: \"0c22f9aa-b3b7-4db2-babb-0f9924d3a2c2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zwfl5" Jul 07 14:49:45.432219 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:49:45.432111 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c22f9aa-b3b7-4db2-babb-0f9924d3a2c2-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zwfl5\" (UID: \"0c22f9aa-b3b7-4db2-babb-0f9924d3a2c2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zwfl5" Jul 07 14:49:45.432219 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:49:45.432142 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c22f9aa-b3b7-4db2-babb-0f9924d3a2c2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zwfl5\" (UID: \"0c22f9aa-b3b7-4db2-babb-0f9924d3a2c2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zwfl5" Jul 07 14:49:45.432219 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:49:45.432186 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c22f9aa-b3b7-4db2-babb-0f9924d3a2c2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zwfl5\" (UID: \"0c22f9aa-b3b7-4db2-babb-0f9924d3a2c2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zwfl5" Jul 07 14:49:45.432413 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:49:45.432219 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c22f9aa-b3b7-4db2-babb-0f9924d3a2c2-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zwfl5\" (UID: \"0c22f9aa-b3b7-4db2-babb-0f9924d3a2c2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zwfl5" Jul 07 14:49:45.432599 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:49:45.432577 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c22f9aa-b3b7-4db2-babb-0f9924d3a2c2-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zwfl5\" (UID: \"0c22f9aa-b3b7-4db2-babb-0f9924d3a2c2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zwfl5" Jul 07 14:49:45.432716 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:49:45.432634 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c22f9aa-b3b7-4db2-babb-0f9924d3a2c2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zwfl5\" (UID: \"0c22f9aa-b3b7-4db2-babb-0f9924d3a2c2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zwfl5" Jul 07 14:49:45.432716 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:49:45.432674 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c22f9aa-b3b7-4db2-babb-0f9924d3a2c2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zwfl5\" (UID: \"0c22f9aa-b3b7-4db2-babb-0f9924d3a2c2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zwfl5" Jul 07 14:49:45.435102 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:49:45.435074 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c22f9aa-b3b7-4db2-babb-0f9924d3a2c2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zwfl5\" (UID: \"0c22f9aa-b3b7-4db2-babb-0f9924d3a2c2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zwfl5" Jul 07 14:49:45.435209 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:49:45.435138 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c22f9aa-b3b7-4db2-babb-0f9924d3a2c2-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zwfl5\" (UID: \"0c22f9aa-b3b7-4db2-babb-0f9924d3a2c2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zwfl5" Jul 07 14:49:45.440610 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:49:45.440584 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m4mj\" (UniqueName: \"kubernetes.io/projected/0c22f9aa-b3b7-4db2-babb-0f9924d3a2c2-kube-api-access-7m4mj\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zwfl5\" (UID: \"0c22f9aa-b3b7-4db2-babb-0f9924d3a2c2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zwfl5" Jul 07 14:49:45.571061 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:49:45.570982 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zwfl5" Jul 07 14:49:45.711325 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:49:45.711301 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-zwfl5"] Jul 07 14:49:45.714170 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:49:45.714143 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c22f9aa_b3b7_4db2_babb_0f9924d3a2c2.slice/crio-0d4d0d93ec5da102d35e36c805e495e3ac8fd9224ce8dd3da3a54f5bed53e38c WatchSource:0}: Error finding container 0d4d0d93ec5da102d35e36c805e495e3ac8fd9224ce8dd3da3a54f5bed53e38c: Status 404 returned error can't find the container with id 0d4d0d93ec5da102d35e36c805e495e3ac8fd9224ce8dd3da3a54f5bed53e38c Jul 07 14:49:45.715905 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:49:45.715887 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 14:49:46.680329 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:49:46.680295 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zwfl5" event={"ID":"0c22f9aa-b3b7-4db2-babb-0f9924d3a2c2","Type":"ContainerStarted","Data":"db095f3eb83d4e7c81ce289a1cde040f3a68cc8d02e0df45e197dd079fd9366b"} Jul 07 14:49:46.680823 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:49:46.680351 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zwfl5" event={"ID":"0c22f9aa-b3b7-4db2-babb-0f9924d3a2c2","Type":"ContainerStarted","Data":"0d4d0d93ec5da102d35e36c805e495e3ac8fd9224ce8dd3da3a54f5bed53e38c"} Jul 07 14:49:46.701232 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:49:46.701181 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zwfl5" podStartSLOduration=1.454154873 podStartE2EDuration="1.701166853s" totalImagesPullingTime="247.01198ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:49:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:49:46.699091936 +0000 UTC m=+928.123278411" watchObservedRunningTime="2026-07-07 14:49:46.701166853 +0000 UTC m=+928.125353372" Jul 07 14:49:55.571733 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:49:55.571695 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zwfl5" Jul 07 14:49:55.572195 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:49:55.571771 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zwfl5" Jul 07 14:49:55.584292 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:49:55.584267 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zwfl5" Jul 07 14:49:55.729177 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:49:55.729149 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zwfl5" Jul 07 14:52:26.839652 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:52:26.839570 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b667b448b-6n6r7"] Jul 07 14:52:26.840131 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:52:26.839775 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-5b667b448b-6n6r7" podUID="85a6e8e4-4b2e-46a0-8589-3436e1b759e1" containerName="manager" containerID="cri-o://21bc79d256ebdf7bde6d00cbd556f8895667d24168b39b3dfad6265f65cc6cc6" gracePeriod=30 Jul 07 14:52:27.085763 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:52:27.085727 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b667b448b-6n6r7" Jul 07 14:52:27.164041 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:52:27.163966 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/85a6e8e4-4b2e-46a0-8589-3436e1b759e1-webhook-cert\") pod \"85a6e8e4-4b2e-46a0-8589-3436e1b759e1\" (UID: \"85a6e8e4-4b2e-46a0-8589-3436e1b759e1\") " Jul 07 14:52:27.164041 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:52:27.163999 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhzz2\" (UniqueName: \"kubernetes.io/projected/85a6e8e4-4b2e-46a0-8589-3436e1b759e1-kube-api-access-rhzz2\") pod \"85a6e8e4-4b2e-46a0-8589-3436e1b759e1\" (UID: \"85a6e8e4-4b2e-46a0-8589-3436e1b759e1\") " Jul 07 14:52:27.165970 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:52:27.165941 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85a6e8e4-4b2e-46a0-8589-3436e1b759e1-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "85a6e8e4-4b2e-46a0-8589-3436e1b759e1" (UID: "85a6e8e4-4b2e-46a0-8589-3436e1b759e1"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:52:27.165970 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:52:27.165953 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85a6e8e4-4b2e-46a0-8589-3436e1b759e1-kube-api-access-rhzz2" (OuterVolumeSpecName: "kube-api-access-rhzz2") pod "85a6e8e4-4b2e-46a0-8589-3436e1b759e1" (UID: "85a6e8e4-4b2e-46a0-8589-3436e1b759e1"). InnerVolumeSpecName "kube-api-access-rhzz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:52:27.265937 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:52:27.265901 2555 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/85a6e8e4-4b2e-46a0-8589-3436e1b759e1-webhook-cert\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:52:27.265937 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:52:27.265939 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhzz2\" (UniqueName: \"kubernetes.io/projected/85a6e8e4-4b2e-46a0-8589-3436e1b759e1-kube-api-access-rhzz2\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 14:52:27.402099 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:52:27.402062 2555 generic.go:356] "Generic (PLEG): container finished" podID="85a6e8e4-4b2e-46a0-8589-3436e1b759e1" containerID="21bc79d256ebdf7bde6d00cbd556f8895667d24168b39b3dfad6265f65cc6cc6" exitCode=0 Jul 07 14:52:27.402295 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:52:27.402124 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b667b448b-6n6r7" Jul 07 14:52:27.402295 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:52:27.402146 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b667b448b-6n6r7" event={"ID":"85a6e8e4-4b2e-46a0-8589-3436e1b759e1","Type":"ContainerDied","Data":"21bc79d256ebdf7bde6d00cbd556f8895667d24168b39b3dfad6265f65cc6cc6"} Jul 07 14:52:27.402295 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:52:27.402235 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b667b448b-6n6r7" event={"ID":"85a6e8e4-4b2e-46a0-8589-3436e1b759e1","Type":"ContainerDied","Data":"55a46c4f41c4bf68271790d2c278f6b95e8deac5833f0c49c4611a8824238a24"} Jul 07 14:52:27.402295 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:52:27.402265 2555 scope.go:117] "RemoveContainer" containerID="21bc79d256ebdf7bde6d00cbd556f8895667d24168b39b3dfad6265f65cc6cc6" Jul 07 14:52:27.411465 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:52:27.411449 2555 scope.go:117] "RemoveContainer" containerID="21bc79d256ebdf7bde6d00cbd556f8895667d24168b39b3dfad6265f65cc6cc6" Jul 07 14:52:27.411728 ip-10-0-136-19 kubenswrapper[2555]: E0707 14:52:27.411706 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21bc79d256ebdf7bde6d00cbd556f8895667d24168b39b3dfad6265f65cc6cc6\": container with ID starting with 21bc79d256ebdf7bde6d00cbd556f8895667d24168b39b3dfad6265f65cc6cc6 not found: ID does not exist" containerID="21bc79d256ebdf7bde6d00cbd556f8895667d24168b39b3dfad6265f65cc6cc6" Jul 07 14:52:27.411845 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:52:27.411733 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21bc79d256ebdf7bde6d00cbd556f8895667d24168b39b3dfad6265f65cc6cc6"} err="failed to get container status \"21bc79d256ebdf7bde6d00cbd556f8895667d24168b39b3dfad6265f65cc6cc6\": rpc error: code = NotFound desc = could not find container \"21bc79d256ebdf7bde6d00cbd556f8895667d24168b39b3dfad6265f65cc6cc6\": container with ID starting with 21bc79d256ebdf7bde6d00cbd556f8895667d24168b39b3dfad6265f65cc6cc6 not found: ID does not exist" Jul 07 14:52:27.425692 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:52:27.425643 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b667b448b-6n6r7"] Jul 07 14:52:27.431052 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:52:27.431032 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5b667b448b-6n6r7"] Jul 07 14:52:28.344892 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:52:28.344857 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5b667b448b-k5gc4"] Jul 07 14:52:28.345264 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:52:28.345185 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85a6e8e4-4b2e-46a0-8589-3436e1b759e1" containerName="manager" Jul 07 14:52:28.345264 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:52:28.345196 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="85a6e8e4-4b2e-46a0-8589-3436e1b759e1" containerName="manager" Jul 07 14:52:28.345264 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:52:28.345249 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="85a6e8e4-4b2e-46a0-8589-3436e1b759e1" containerName="manager" Jul 07 14:52:28.349012 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:52:28.348996 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b667b448b-k5gc4" Jul 07 14:52:28.354000 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:52:28.353976 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 07 14:52:28.354132 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:52:28.354109 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 07 14:52:28.354928 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:52:28.354910 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mhvhq\"" Jul 07 14:52:28.373140 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:52:28.373114 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b667b448b-k5gc4"] Jul 07 14:52:28.375471 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:52:28.375447 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bb97ab4b-9777-44dc-9ba6-646e64b881ea-webhook-cert\") pod \"maas-controller-5b667b448b-k5gc4\" (UID: \"bb97ab4b-9777-44dc-9ba6-646e64b881ea\") " pod="opendatahub/maas-controller-5b667b448b-k5gc4" Jul 07 14:52:28.375595 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:52:28.375499 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zwrs\" (UniqueName: \"kubernetes.io/projected/bb97ab4b-9777-44dc-9ba6-646e64b881ea-kube-api-access-8zwrs\") pod \"maas-controller-5b667b448b-k5gc4\" (UID: \"bb97ab4b-9777-44dc-9ba6-646e64b881ea\") " pod="opendatahub/maas-controller-5b667b448b-k5gc4" Jul 07 14:52:28.476525 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:52:28.476488 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bb97ab4b-9777-44dc-9ba6-646e64b881ea-webhook-cert\") pod \"maas-controller-5b667b448b-k5gc4\" (UID: \"bb97ab4b-9777-44dc-9ba6-646e64b881ea\") " pod="opendatahub/maas-controller-5b667b448b-k5gc4" Jul 07 14:52:28.476689 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:52:28.476532 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zwrs\" (UniqueName: \"kubernetes.io/projected/bb97ab4b-9777-44dc-9ba6-646e64b881ea-kube-api-access-8zwrs\") pod \"maas-controller-5b667b448b-k5gc4\" (UID: \"bb97ab4b-9777-44dc-9ba6-646e64b881ea\") " pod="opendatahub/maas-controller-5b667b448b-k5gc4" Jul 07 14:52:28.478956 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:52:28.478933 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bb97ab4b-9777-44dc-9ba6-646e64b881ea-webhook-cert\") pod \"maas-controller-5b667b448b-k5gc4\" (UID: \"bb97ab4b-9777-44dc-9ba6-646e64b881ea\") " pod="opendatahub/maas-controller-5b667b448b-k5gc4" Jul 07 14:52:28.487224 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:52:28.487197 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zwrs\" (UniqueName: \"kubernetes.io/projected/bb97ab4b-9777-44dc-9ba6-646e64b881ea-kube-api-access-8zwrs\") pod \"maas-controller-5b667b448b-k5gc4\" (UID: \"bb97ab4b-9777-44dc-9ba6-646e64b881ea\") " pod="opendatahub/maas-controller-5b667b448b-k5gc4" Jul 07 14:52:28.658352 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:52:28.658263 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b667b448b-k5gc4" Jul 07 14:52:28.789657 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:52:28.789628 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b667b448b-k5gc4"] Jul 07 14:52:28.792083 ip-10-0-136-19 kubenswrapper[2555]: W0707 14:52:28.792056 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb97ab4b_9777_44dc_9ba6_646e64b881ea.slice/crio-58762d0131b9e01b0dc152adc0aa502f90c4d33241d32d5b16bb5c7bac597e75 WatchSource:0}: Error finding container 58762d0131b9e01b0dc152adc0aa502f90c4d33241d32d5b16bb5c7bac597e75: Status 404 returned error can't find the container with id 58762d0131b9e01b0dc152adc0aa502f90c4d33241d32d5b16bb5c7bac597e75 Jul 07 14:52:29.042107 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:52:29.042014 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85a6e8e4-4b2e-46a0-8589-3436e1b759e1" path="/var/lib/kubelet/pods/85a6e8e4-4b2e-46a0-8589-3436e1b759e1/volumes" Jul 07 14:52:29.416309 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:52:29.416272 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b667b448b-k5gc4" event={"ID":"bb97ab4b-9777-44dc-9ba6-646e64b881ea","Type":"ContainerStarted","Data":"14a1bfaff0afec5b5a6510228b88654e10aad576fab005be8da31e4656f2ef5c"} Jul 07 14:52:29.416791 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:52:29.416329 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b667b448b-k5gc4" event={"ID":"bb97ab4b-9777-44dc-9ba6-646e64b881ea","Type":"ContainerStarted","Data":"58762d0131b9e01b0dc152adc0aa502f90c4d33241d32d5b16bb5c7bac597e75"} Jul 07 14:52:29.416791 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:52:29.416442 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5b667b448b-k5gc4" Jul 07 14:52:29.445104 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:52:29.445061 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5b667b448b-k5gc4" podStartSLOduration=0.966295819 podStartE2EDuration="1.445045799s" totalImagesPullingTime="478.74998ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:52:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:52:29.444285214 +0000 UTC m=+1090.868471686" watchObservedRunningTime="2026-07-07 14:52:29.445045799 +0000 UTC m=+1090.869232272" Jul 07 14:52:40.426584 ip-10-0-136-19 kubenswrapper[2555]: I0707 14:52:40.426556 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5b667b448b-k5gc4" Jul 07 15:04:25.053462 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:25.053422 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n"] Jul 07 15:04:25.057344 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:25.057321 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n" Jul 07 15:04:25.060175 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:25.060154 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c722dcc9-openshift-default-dockercfg-rwl65\"" Jul 07 15:04:25.081175 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:25.081151 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n"] Jul 07 15:04:25.101152 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:25.101121 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a806885a-99e7-45c0-9990-51932972f7ac-workload-socket\") pod \"e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n\" (UID: \"a806885a-99e7-45c0-9990-51932972f7ac\") " pod="openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n" Jul 07 15:04:25.101323 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:25.101160 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a806885a-99e7-45c0-9990-51932972f7ac-istio-envoy\") pod \"e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n\" (UID: \"a806885a-99e7-45c0-9990-51932972f7ac\") " pod="openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n" Jul 07 15:04:25.101323 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:25.101226 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a806885a-99e7-45c0-9990-51932972f7ac-istio-podinfo\") pod \"e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n\" (UID: \"a806885a-99e7-45c0-9990-51932972f7ac\") " pod="openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n" Jul 07 15:04:25.101323 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:25.101246 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a806885a-99e7-45c0-9990-51932972f7ac-istio-token\") pod \"e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n\" (UID: \"a806885a-99e7-45c0-9990-51932972f7ac\") " pod="openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n" Jul 07 15:04:25.101323 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:25.101276 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a806885a-99e7-45c0-9990-51932972f7ac-istiod-ca-cert\") pod \"e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n\" (UID: \"a806885a-99e7-45c0-9990-51932972f7ac\") " pod="openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n" Jul 07 15:04:25.101323 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:25.101304 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fz4k\" (UniqueName: \"kubernetes.io/projected/a806885a-99e7-45c0-9990-51932972f7ac-kube-api-access-6fz4k\") pod \"e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n\" (UID: \"a806885a-99e7-45c0-9990-51932972f7ac\") " pod="openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n" Jul 07 15:04:25.102034 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:25.101425 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a806885a-99e7-45c0-9990-51932972f7ac-workload-certs\") pod \"e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n\" (UID: \"a806885a-99e7-45c0-9990-51932972f7ac\") " pod="openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n" Jul 07 15:04:25.102034 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:25.101490 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a806885a-99e7-45c0-9990-51932972f7ac-istio-data\") pod \"e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n\" (UID: \"a806885a-99e7-45c0-9990-51932972f7ac\") " pod="openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n" Jul 07 15:04:25.102034 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:25.101533 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a806885a-99e7-45c0-9990-51932972f7ac-credential-socket\") pod \"e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n\" (UID: \"a806885a-99e7-45c0-9990-51932972f7ac\") " pod="openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n" Jul 07 15:04:25.202728 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:25.202694 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a806885a-99e7-45c0-9990-51932972f7ac-workload-certs\") pod \"e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n\" (UID: \"a806885a-99e7-45c0-9990-51932972f7ac\") " pod="openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n" Jul 07 15:04:25.202921 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:25.202737 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a806885a-99e7-45c0-9990-51932972f7ac-istio-data\") pod \"e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n\" (UID: \"a806885a-99e7-45c0-9990-51932972f7ac\") " pod="openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n" Jul 07 15:04:25.202921 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:25.202799 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a806885a-99e7-45c0-9990-51932972f7ac-credential-socket\") pod \"e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n\" (UID: \"a806885a-99e7-45c0-9990-51932972f7ac\") " pod="openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n" Jul 07 15:04:25.202921 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:25.202832 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a806885a-99e7-45c0-9990-51932972f7ac-workload-socket\") pod \"e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n\" (UID: \"a806885a-99e7-45c0-9990-51932972f7ac\") " pod="openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n" Jul 07 15:04:25.202921 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:25.202863 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a806885a-99e7-45c0-9990-51932972f7ac-istio-envoy\") pod \"e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n\" (UID: \"a806885a-99e7-45c0-9990-51932972f7ac\") " pod="openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n" Jul 07 15:04:25.202921 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:25.202913 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a806885a-99e7-45c0-9990-51932972f7ac-istio-podinfo\") pod \"e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n\" (UID: \"a806885a-99e7-45c0-9990-51932972f7ac\") " pod="openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n" Jul 07 15:04:25.203194 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:25.202936 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a806885a-99e7-45c0-9990-51932972f7ac-istio-token\") pod \"e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n\" (UID: \"a806885a-99e7-45c0-9990-51932972f7ac\") " pod="openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n" Jul 07 15:04:25.203194 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:25.202965 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a806885a-99e7-45c0-9990-51932972f7ac-istiod-ca-cert\") pod \"e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n\" (UID: \"a806885a-99e7-45c0-9990-51932972f7ac\") " pod="openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n" Jul 07 15:04:25.203194 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:25.202994 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fz4k\" (UniqueName: \"kubernetes.io/projected/a806885a-99e7-45c0-9990-51932972f7ac-kube-api-access-6fz4k\") pod \"e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n\" (UID: \"a806885a-99e7-45c0-9990-51932972f7ac\") " pod="openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n" Jul 07 15:04:25.203194 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:25.203156 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a806885a-99e7-45c0-9990-51932972f7ac-workload-certs\") pod \"e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n\" (UID: \"a806885a-99e7-45c0-9990-51932972f7ac\") " pod="openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n" Jul 07 15:04:25.203407 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:25.203204 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a806885a-99e7-45c0-9990-51932972f7ac-credential-socket\") pod \"e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n\" (UID: \"a806885a-99e7-45c0-9990-51932972f7ac\") " pod="openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n" Jul 07 15:04:25.203407 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:25.203253 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a806885a-99e7-45c0-9990-51932972f7ac-istio-data\") pod \"e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n\" (UID: \"a806885a-99e7-45c0-9990-51932972f7ac\") " pod="openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n" Jul 07 15:04:25.203407 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:25.203315 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a806885a-99e7-45c0-9990-51932972f7ac-workload-socket\") pod \"e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n\" (UID: \"a806885a-99e7-45c0-9990-51932972f7ac\") " pod="openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n" Jul 07 15:04:25.203789 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:25.203769 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a806885a-99e7-45c0-9990-51932972f7ac-istiod-ca-cert\") pod \"e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n\" (UID: \"a806885a-99e7-45c0-9990-51932972f7ac\") " pod="openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n" Jul 07 15:04:25.205321 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:25.205299 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a806885a-99e7-45c0-9990-51932972f7ac-istio-envoy\") pod \"e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n\" (UID: \"a806885a-99e7-45c0-9990-51932972f7ac\") " pod="openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n" Jul 07 15:04:25.205515 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:25.205496 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a806885a-99e7-45c0-9990-51932972f7ac-istio-podinfo\") pod \"e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n\" (UID: \"a806885a-99e7-45c0-9990-51932972f7ac\") " pod="openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n" Jul 07 15:04:25.211728 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:25.211699 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a806885a-99e7-45c0-9990-51932972f7ac-istio-token\") pod \"e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n\" (UID: \"a806885a-99e7-45c0-9990-51932972f7ac\") " pod="openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n" Jul 07 15:04:25.211879 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:25.211838 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fz4k\" (UniqueName: \"kubernetes.io/projected/a806885a-99e7-45c0-9990-51932972f7ac-kube-api-access-6fz4k\") pod \"e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n\" (UID: \"a806885a-99e7-45c0-9990-51932972f7ac\") " pod="openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n" Jul 07 15:04:25.368173 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:25.368124 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n" Jul 07 15:04:25.508529 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:25.508504 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n"] Jul 07 15:04:25.511122 ip-10-0-136-19 kubenswrapper[2555]: W0707 15:04:25.511090 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda806885a_99e7_45c0_9990_51932972f7ac.slice/crio-9a5ccfc8b5d15525a7c396d893816415a07349451fa480b0f76606ec231c7565 WatchSource:0}: Error finding container 9a5ccfc8b5d15525a7c396d893816415a07349451fa480b0f76606ec231c7565: Status 404 returned error can't find the container with id 9a5ccfc8b5d15525a7c396d893816415a07349451fa480b0f76606ec231c7565 Jul 07 15:04:25.513226 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:25.513191 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:04:25.513327 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:25.513268 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:04:25.513327 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:25.513307 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:04:25.557452 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:25.557424 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n" event={"ID":"a806885a-99e7-45c0-9990-51932972f7ac","Type":"ContainerStarted","Data":"9a5ccfc8b5d15525a7c396d893816415a07349451fa480b0f76606ec231c7565"} Jul 07 15:04:26.338428 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:26.338396 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-c722dcc9-c8d58d64c-x7tkf"] Jul 07 15:04:26.342207 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:26.342181 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-c722dcc9-c8d58d64c-x7tkf" Jul 07 15:04:26.345101 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:26.345078 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-c722dcc9-serving-cert\"" Jul 07 15:04:26.354956 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:26.354932 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-c722dcc9-c8d58d64c-x7tkf"] Jul 07 15:04:26.414386 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:26.414348 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02c20c14-d7b8-4c9e-9d78-ceeaabe1c940-openshift-service-ca\") pod \"maas-api-e2e-ait-c722dcc9-c8d58d64c-x7tkf\" (UID: \"02c20c14-d7b8-4c9e-9d78-ceeaabe1c940\") " pod="opendatahub/maas-api-e2e-ait-c722dcc9-c8d58d64c-x7tkf" Jul 07 15:04:26.414575 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:26.414407 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02c20c14-d7b8-4c9e-9d78-ceeaabe1c940-maas-api-tls\") pod \"maas-api-e2e-ait-c722dcc9-c8d58d64c-x7tkf\" (UID: \"02c20c14-d7b8-4c9e-9d78-ceeaabe1c940\") " pod="opendatahub/maas-api-e2e-ait-c722dcc9-c8d58d64c-x7tkf" Jul 07 15:04:26.414575 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:26.414509 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl6h7\" (UniqueName: \"kubernetes.io/projected/02c20c14-d7b8-4c9e-9d78-ceeaabe1c940-kube-api-access-zl6h7\") pod \"maas-api-e2e-ait-c722dcc9-c8d58d64c-x7tkf\" (UID: \"02c20c14-d7b8-4c9e-9d78-ceeaabe1c940\") " pod="opendatahub/maas-api-e2e-ait-c722dcc9-c8d58d64c-x7tkf" Jul 07 15:04:26.515666 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:26.515631 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl6h7\" (UniqueName: \"kubernetes.io/projected/02c20c14-d7b8-4c9e-9d78-ceeaabe1c940-kube-api-access-zl6h7\") pod \"maas-api-e2e-ait-c722dcc9-c8d58d64c-x7tkf\" (UID: \"02c20c14-d7b8-4c9e-9d78-ceeaabe1c940\") " pod="opendatahub/maas-api-e2e-ait-c722dcc9-c8d58d64c-x7tkf" Jul 07 15:04:26.515880 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:26.515698 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02c20c14-d7b8-4c9e-9d78-ceeaabe1c940-openshift-service-ca\") pod \"maas-api-e2e-ait-c722dcc9-c8d58d64c-x7tkf\" (UID: \"02c20c14-d7b8-4c9e-9d78-ceeaabe1c940\") " pod="opendatahub/maas-api-e2e-ait-c722dcc9-c8d58d64c-x7tkf" Jul 07 15:04:26.515880 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:26.515741 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02c20c14-d7b8-4c9e-9d78-ceeaabe1c940-maas-api-tls\") pod \"maas-api-e2e-ait-c722dcc9-c8d58d64c-x7tkf\" (UID: \"02c20c14-d7b8-4c9e-9d78-ceeaabe1c940\") " pod="opendatahub/maas-api-e2e-ait-c722dcc9-c8d58d64c-x7tkf" Jul 07 15:04:26.516333 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:26.516315 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02c20c14-d7b8-4c9e-9d78-ceeaabe1c940-openshift-service-ca\") pod \"maas-api-e2e-ait-c722dcc9-c8d58d64c-x7tkf\" (UID: \"02c20c14-d7b8-4c9e-9d78-ceeaabe1c940\") " pod="opendatahub/maas-api-e2e-ait-c722dcc9-c8d58d64c-x7tkf" Jul 07 15:04:26.518133 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:26.518117 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02c20c14-d7b8-4c9e-9d78-ceeaabe1c940-maas-api-tls\") pod \"maas-api-e2e-ait-c722dcc9-c8d58d64c-x7tkf\" (UID: \"02c20c14-d7b8-4c9e-9d78-ceeaabe1c940\") " pod="opendatahub/maas-api-e2e-ait-c722dcc9-c8d58d64c-x7tkf" Jul 07 15:04:26.525990 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:26.525965 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl6h7\" (UniqueName: \"kubernetes.io/projected/02c20c14-d7b8-4c9e-9d78-ceeaabe1c940-kube-api-access-zl6h7\") pod \"maas-api-e2e-ait-c722dcc9-c8d58d64c-x7tkf\" (UID: \"02c20c14-d7b8-4c9e-9d78-ceeaabe1c940\") " pod="opendatahub/maas-api-e2e-ait-c722dcc9-c8d58d64c-x7tkf" Jul 07 15:04:26.563814 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:26.563783 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n" event={"ID":"a806885a-99e7-45c0-9990-51932972f7ac","Type":"ContainerStarted","Data":"c301e8d5bb7ec343cba5d67782d874d9af83cec0af85114d2c83eccb219fe849"} Jul 07 15:04:26.600290 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:26.600178 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n" podStartSLOduration=1.600161842 podStartE2EDuration="1.600161842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:04:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:04:26.598390952 +0000 UTC m=+1808.022577428" watchObservedRunningTime="2026-07-07 15:04:26.600161842 +0000 UTC m=+1808.024348314" Jul 07 15:04:26.654150 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:26.654116 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-c722dcc9-c8d58d64c-x7tkf" Jul 07 15:04:26.802670 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:26.802636 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-c722dcc9-c8d58d64c-x7tkf"] Jul 07 15:04:26.805311 ip-10-0-136-19 kubenswrapper[2555]: W0707 15:04:26.805284 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02c20c14_d7b8_4c9e_9d78_ceeaabe1c940.slice/crio-28b53912075d08d72a1ce6cc9ed623faed852f2a891b9231b07b6257fffb32df WatchSource:0}: Error finding container 28b53912075d08d72a1ce6cc9ed623faed852f2a891b9231b07b6257fffb32df: Status 404 returned error can't find the container with id 28b53912075d08d72a1ce6cc9ed623faed852f2a891b9231b07b6257fffb32df Jul 07 15:04:26.806691 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:26.806674 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 15:04:27.368569 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:27.368537 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n" Jul 07 15:04:27.373167 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:27.373142 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n" Jul 07 15:04:27.570327 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:27.570233 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-c722dcc9-c8d58d64c-x7tkf" event={"ID":"02c20c14-d7b8-4c9e-9d78-ceeaabe1c940","Type":"ContainerStarted","Data":"24fe9ae3bc04f2b1fcb0247d79b1bc39f8295015a10458d526ef08cfcd81aa23"} Jul 07 15:04:27.570477 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:27.570325 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n" Jul 07 15:04:27.570477 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:27.570359 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-c722dcc9-c8d58d64c-x7tkf" event={"ID":"02c20c14-d7b8-4c9e-9d78-ceeaabe1c940","Type":"ContainerStarted","Data":"28b53912075d08d72a1ce6cc9ed623faed852f2a891b9231b07b6257fffb32df"} Jul 07 15:04:27.570628 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:27.570609 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-c722dcc9-c8d58d64c-x7tkf" Jul 07 15:04:27.571363 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:27.571341 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n" Jul 07 15:04:27.590287 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:27.590238 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-c722dcc9-c8d58d64c-x7tkf" podStartSLOduration=1.137057297 podStartE2EDuration="1.59022538s" totalImagesPullingTime="453.168083ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:04:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:04:27.588114008 +0000 UTC m=+1809.012300485" watchObservedRunningTime="2026-07-07 15:04:27.59022538 +0000 UTC m=+1809.014411854" Jul 07 15:04:31.620559 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:31.620525 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n"] Jul 07 15:04:31.620968 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:31.620732 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n" podUID="a806885a-99e7-45c0-9990-51932972f7ac" containerName="istio-proxy" containerID="cri-o://c301e8d5bb7ec343cba5d67782d874d9af83cec0af85114d2c83eccb219fe849" gracePeriod=30 Jul 07 15:04:31.899958 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:31.899859 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-c722dcc9-c8d58d64c-x7tkf"] Jul 07 15:04:31.900422 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:31.900386 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-c722dcc9-c8d58d64c-x7tkf" podUID="02c20c14-d7b8-4c9e-9d78-ceeaabe1c940" containerName="maas-api" containerID="cri-o://24fe9ae3bc04f2b1fcb0247d79b1bc39f8295015a10458d526ef08cfcd81aa23" gracePeriod=30 Jul 07 15:04:32.191006 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:32.190971 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-c722dcc9-c8d58d64c-x7tkf" Jul 07 15:04:32.277555 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:32.277508 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zl6h7\" (UniqueName: \"kubernetes.io/projected/02c20c14-d7b8-4c9e-9d78-ceeaabe1c940-kube-api-access-zl6h7\") pod \"02c20c14-d7b8-4c9e-9d78-ceeaabe1c940\" (UID: \"02c20c14-d7b8-4c9e-9d78-ceeaabe1c940\") " Jul 07 15:04:32.277818 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:32.277566 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02c20c14-d7b8-4c9e-9d78-ceeaabe1c940-openshift-service-ca\") pod \"02c20c14-d7b8-4c9e-9d78-ceeaabe1c940\" (UID: \"02c20c14-d7b8-4c9e-9d78-ceeaabe1c940\") " Jul 07 15:04:32.277818 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:32.277589 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02c20c14-d7b8-4c9e-9d78-ceeaabe1c940-maas-api-tls\") pod \"02c20c14-d7b8-4c9e-9d78-ceeaabe1c940\" (UID: \"02c20c14-d7b8-4c9e-9d78-ceeaabe1c940\") " Jul 07 15:04:32.277999 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:32.277973 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02c20c14-d7b8-4c9e-9d78-ceeaabe1c940-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "02c20c14-d7b8-4c9e-9d78-ceeaabe1c940" (UID: "02c20c14-d7b8-4c9e-9d78-ceeaabe1c940"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:04:32.279816 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:32.279786 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02c20c14-d7b8-4c9e-9d78-ceeaabe1c940-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "02c20c14-d7b8-4c9e-9d78-ceeaabe1c940" (UID: "02c20c14-d7b8-4c9e-9d78-ceeaabe1c940"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:04:32.279929 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:32.279823 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02c20c14-d7b8-4c9e-9d78-ceeaabe1c940-kube-api-access-zl6h7" (OuterVolumeSpecName: "kube-api-access-zl6h7") pod "02c20c14-d7b8-4c9e-9d78-ceeaabe1c940" (UID: "02c20c14-d7b8-4c9e-9d78-ceeaabe1c940"). InnerVolumeSpecName "kube-api-access-zl6h7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:04:32.378323 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:32.378288 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02c20c14-d7b8-4c9e-9d78-ceeaabe1c940-openshift-service-ca\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:04:32.378323 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:32.378319 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02c20c14-d7b8-4c9e-9d78-ceeaabe1c940-maas-api-tls\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:04:32.378323 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:32.378330 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl6h7\" (UniqueName: \"kubernetes.io/projected/02c20c14-d7b8-4c9e-9d78-ceeaabe1c940-kube-api-access-zl6h7\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:04:32.600860 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:32.600819 2555 generic.go:356] "Generic (PLEG): container finished" podID="02c20c14-d7b8-4c9e-9d78-ceeaabe1c940" containerID="24fe9ae3bc04f2b1fcb0247d79b1bc39f8295015a10458d526ef08cfcd81aa23" exitCode=0 Jul 07 15:04:32.601062 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:32.600882 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-c722dcc9-c8d58d64c-x7tkf" Jul 07 15:04:32.601062 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:32.600898 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-c722dcc9-c8d58d64c-x7tkf" event={"ID":"02c20c14-d7b8-4c9e-9d78-ceeaabe1c940","Type":"ContainerDied","Data":"24fe9ae3bc04f2b1fcb0247d79b1bc39f8295015a10458d526ef08cfcd81aa23"} Jul 07 15:04:32.601062 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:32.600987 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-c722dcc9-c8d58d64c-x7tkf" event={"ID":"02c20c14-d7b8-4c9e-9d78-ceeaabe1c940","Type":"ContainerDied","Data":"28b53912075d08d72a1ce6cc9ed623faed852f2a891b9231b07b6257fffb32df"} Jul 07 15:04:32.601062 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:32.601011 2555 scope.go:117] "RemoveContainer" containerID="24fe9ae3bc04f2b1fcb0247d79b1bc39f8295015a10458d526ef08cfcd81aa23" Jul 07 15:04:32.610313 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:32.610294 2555 scope.go:117] "RemoveContainer" containerID="24fe9ae3bc04f2b1fcb0247d79b1bc39f8295015a10458d526ef08cfcd81aa23" Jul 07 15:04:32.610598 ip-10-0-136-19 kubenswrapper[2555]: E0707 15:04:32.610580 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24fe9ae3bc04f2b1fcb0247d79b1bc39f8295015a10458d526ef08cfcd81aa23\": container with ID starting with 24fe9ae3bc04f2b1fcb0247d79b1bc39f8295015a10458d526ef08cfcd81aa23 not found: ID does not exist" containerID="24fe9ae3bc04f2b1fcb0247d79b1bc39f8295015a10458d526ef08cfcd81aa23" Jul 07 15:04:32.610647 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:32.610606 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24fe9ae3bc04f2b1fcb0247d79b1bc39f8295015a10458d526ef08cfcd81aa23"} err="failed to get container status \"24fe9ae3bc04f2b1fcb0247d79b1bc39f8295015a10458d526ef08cfcd81aa23\": rpc error: code = NotFound desc = could not find container \"24fe9ae3bc04f2b1fcb0247d79b1bc39f8295015a10458d526ef08cfcd81aa23\": container with ID starting with 24fe9ae3bc04f2b1fcb0247d79b1bc39f8295015a10458d526ef08cfcd81aa23 not found: ID does not exist" Jul 07 15:04:32.622333 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:32.622308 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-c722dcc9-c8d58d64c-x7tkf"] Jul 07 15:04:32.626784 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:32.626743 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-c722dcc9-c8d58d64c-x7tkf"] Jul 07 15:04:33.062960 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:33.062910 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02c20c14-d7b8-4c9e-9d78-ceeaabe1c940" path="/var/lib/kubelet/pods/02c20c14-d7b8-4c9e-9d78-ceeaabe1c940/volumes" Jul 07 15:04:36.475611 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:36.475585 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n" Jul 07 15:04:36.521789 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:36.521676 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a806885a-99e7-45c0-9990-51932972f7ac-istiod-ca-cert\") pod \"a806885a-99e7-45c0-9990-51932972f7ac\" (UID: \"a806885a-99e7-45c0-9990-51932972f7ac\") " Jul 07 15:04:36.521789 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:36.521711 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a806885a-99e7-45c0-9990-51932972f7ac-istio-data\") pod \"a806885a-99e7-45c0-9990-51932972f7ac\" (UID: \"a806885a-99e7-45c0-9990-51932972f7ac\") " Jul 07 15:04:36.521789 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:36.521778 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a806885a-99e7-45c0-9990-51932972f7ac-workload-certs\") pod \"a806885a-99e7-45c0-9990-51932972f7ac\" (UID: \"a806885a-99e7-45c0-9990-51932972f7ac\") " Jul 07 15:04:36.522104 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:36.521796 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a806885a-99e7-45c0-9990-51932972f7ac-credential-socket\") pod \"a806885a-99e7-45c0-9990-51932972f7ac\" (UID: \"a806885a-99e7-45c0-9990-51932972f7ac\") " Jul 07 15:04:36.522104 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:36.521851 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a806885a-99e7-45c0-9990-51932972f7ac-workload-socket\") pod \"a806885a-99e7-45c0-9990-51932972f7ac\" (UID: \"a806885a-99e7-45c0-9990-51932972f7ac\") " Jul 07 15:04:36.522104 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:36.521907 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a806885a-99e7-45c0-9990-51932972f7ac-istio-podinfo\") pod \"a806885a-99e7-45c0-9990-51932972f7ac\" (UID: \"a806885a-99e7-45c0-9990-51932972f7ac\") " Jul 07 15:04:36.522104 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:36.521950 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a806885a-99e7-45c0-9990-51932972f7ac-istio-envoy\") pod \"a806885a-99e7-45c0-9990-51932972f7ac\" (UID: \"a806885a-99e7-45c0-9990-51932972f7ac\") " Jul 07 15:04:36.522104 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:36.521975 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fz4k\" (UniqueName: \"kubernetes.io/projected/a806885a-99e7-45c0-9990-51932972f7ac-kube-api-access-6fz4k\") pod \"a806885a-99e7-45c0-9990-51932972f7ac\" (UID: \"a806885a-99e7-45c0-9990-51932972f7ac\") " Jul 07 15:04:36.522104 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:36.522002 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a806885a-99e7-45c0-9990-51932972f7ac-istio-token\") pod \"a806885a-99e7-45c0-9990-51932972f7ac\" (UID: \"a806885a-99e7-45c0-9990-51932972f7ac\") " Jul 07 15:04:36.522427 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:36.522114 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a806885a-99e7-45c0-9990-51932972f7ac-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a806885a-99e7-45c0-9990-51932972f7ac" (UID: "a806885a-99e7-45c0-9990-51932972f7ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:04:36.522427 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:36.522274 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a806885a-99e7-45c0-9990-51932972f7ac-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a806885a-99e7-45c0-9990-51932972f7ac" (UID: "a806885a-99e7-45c0-9990-51932972f7ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:04:36.522427 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:36.522378 2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a806885a-99e7-45c0-9990-51932972f7ac-workload-certs\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:04:36.522427 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:36.522397 2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a806885a-99e7-45c0-9990-51932972f7ac-credential-socket\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:04:36.522644 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:36.522489 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a806885a-99e7-45c0-9990-51932972f7ac-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a806885a-99e7-45c0-9990-51932972f7ac" (UID: "a806885a-99e7-45c0-9990-51932972f7ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:04:36.522644 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:36.522505 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a806885a-99e7-45c0-9990-51932972f7ac-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a806885a-99e7-45c0-9990-51932972f7ac" (UID: "a806885a-99e7-45c0-9990-51932972f7ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:04:36.522772 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:36.522647 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a806885a-99e7-45c0-9990-51932972f7ac-istio-data" (OuterVolumeSpecName: "istio-data") pod "a806885a-99e7-45c0-9990-51932972f7ac" (UID: "a806885a-99e7-45c0-9990-51932972f7ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:04:36.524533 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:36.524506 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a806885a-99e7-45c0-9990-51932972f7ac-kube-api-access-6fz4k" (OuterVolumeSpecName: "kube-api-access-6fz4k") pod "a806885a-99e7-45c0-9990-51932972f7ac" (UID: "a806885a-99e7-45c0-9990-51932972f7ac"). InnerVolumeSpecName "kube-api-access-6fz4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:04:36.524533 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:36.524522 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a806885a-99e7-45c0-9990-51932972f7ac-istio-token" (OuterVolumeSpecName: "istio-token") pod "a806885a-99e7-45c0-9990-51932972f7ac" (UID: "a806885a-99e7-45c0-9990-51932972f7ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:04:36.524736 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:36.524665 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a806885a-99e7-45c0-9990-51932972f7ac-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a806885a-99e7-45c0-9990-51932972f7ac" (UID: "a806885a-99e7-45c0-9990-51932972f7ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:04:36.524736 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:36.524665 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a806885a-99e7-45c0-9990-51932972f7ac-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a806885a-99e7-45c0-9990-51932972f7ac" (UID: "a806885a-99e7-45c0-9990-51932972f7ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 15:04:36.622586 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:36.622555 2555 generic.go:356] "Generic (PLEG): container finished" podID="a806885a-99e7-45c0-9990-51932972f7ac" containerID="c301e8d5bb7ec343cba5d67782d874d9af83cec0af85114d2c83eccb219fe849" exitCode=0 Jul 07 15:04:36.622831 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:36.622606 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n" Jul 07 15:04:36.622831 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:36.622648 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n" event={"ID":"a806885a-99e7-45c0-9990-51932972f7ac","Type":"ContainerDied","Data":"c301e8d5bb7ec343cba5d67782d874d9af83cec0af85114d2c83eccb219fe849"} Jul 07 15:04:36.622831 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:36.622703 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n" event={"ID":"a806885a-99e7-45c0-9990-51932972f7ac","Type":"ContainerDied","Data":"9a5ccfc8b5d15525a7c396d893816415a07349451fa480b0f76606ec231c7565"} Jul 07 15:04:36.622831 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:36.622727 2555 scope.go:117] "RemoveContainer" containerID="c301e8d5bb7ec343cba5d67782d874d9af83cec0af85114d2c83eccb219fe849" Jul 07 15:04:36.623044 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:36.622980 2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a806885a-99e7-45c0-9990-51932972f7ac-istio-podinfo\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:04:36.623044 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:36.623004 2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a806885a-99e7-45c0-9990-51932972f7ac-istio-envoy\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:04:36.623044 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:36.623020 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fz4k\" (UniqueName: \"kubernetes.io/projected/a806885a-99e7-45c0-9990-51932972f7ac-kube-api-access-6fz4k\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:04:36.623044 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:36.623036 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a806885a-99e7-45c0-9990-51932972f7ac-istio-token\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:04:36.623222 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:36.623049 2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a806885a-99e7-45c0-9990-51932972f7ac-istiod-ca-cert\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:04:36.623222 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:36.623061 2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a806885a-99e7-45c0-9990-51932972f7ac-istio-data\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:04:36.623222 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:36.623075 2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a806885a-99e7-45c0-9990-51932972f7ac-workload-socket\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:04:36.632540 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:36.632516 2555 scope.go:117] "RemoveContainer" containerID="c301e8d5bb7ec343cba5d67782d874d9af83cec0af85114d2c83eccb219fe849" Jul 07 15:04:36.632875 ip-10-0-136-19 kubenswrapper[2555]: E0707 15:04:36.632857 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c301e8d5bb7ec343cba5d67782d874d9af83cec0af85114d2c83eccb219fe849\": container with ID starting with c301e8d5bb7ec343cba5d67782d874d9af83cec0af85114d2c83eccb219fe849 not found: ID does not exist" containerID="c301e8d5bb7ec343cba5d67782d874d9af83cec0af85114d2c83eccb219fe849" Jul 07 15:04:36.632927 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:36.632884 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c301e8d5bb7ec343cba5d67782d874d9af83cec0af85114d2c83eccb219fe849"} err="failed to get container status \"c301e8d5bb7ec343cba5d67782d874d9af83cec0af85114d2c83eccb219fe849\": rpc error: code = NotFound desc = could not find container \"c301e8d5bb7ec343cba5d67782d874d9af83cec0af85114d2c83eccb219fe849\": container with ID starting with c301e8d5bb7ec343cba5d67782d874d9af83cec0af85114d2c83eccb219fe849 not found: ID does not exist" Jul 07 15:04:36.646997 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:36.646965 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n"] Jul 07 15:04:36.651035 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:36.651004 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c722dcc9-openshift-default-86966c9f6f-7kd9n"] Jul 07 15:04:37.042841 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:37.042801 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a806885a-99e7-45c0-9990-51932972f7ac" path="/var/lib/kubelet/pods/a806885a-99e7-45c0-9990-51932972f7ac/volumes" Jul 07 15:04:38.037622 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:38.037588 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr"] Jul 07 15:04:38.038033 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:38.037989 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02c20c14-d7b8-4c9e-9d78-ceeaabe1c940" containerName="maas-api" Jul 07 15:04:38.038033 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:38.038002 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="02c20c14-d7b8-4c9e-9d78-ceeaabe1c940" containerName="maas-api" Jul 07 15:04:38.038134 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:38.038040 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a806885a-99e7-45c0-9990-51932972f7ac" containerName="istio-proxy" Jul 07 15:04:38.038134 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:38.038046 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a806885a-99e7-45c0-9990-51932972f7ac" containerName="istio-proxy" Jul 07 15:04:38.038134 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:38.038103 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a806885a-99e7-45c0-9990-51932972f7ac" containerName="istio-proxy" Jul 07 15:04:38.038134 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:38.038112 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="02c20c14-d7b8-4c9e-9d78-ceeaabe1c940" containerName="maas-api" Jul 07 15:04:38.042792 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:38.042741 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr" Jul 07 15:04:38.045742 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:38.045718 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e5b77db4-openshift-default-dockercfg-rb52l\"" Jul 07 15:04:38.058349 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:38.058313 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr"] Jul 07 15:04:38.142246 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:38.142209 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-credential-socket\") pod \"e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr\" (UID: \"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec\") " pod="openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr" Jul 07 15:04:38.142246 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:38.142246 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-istio-podinfo\") pod \"e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr\" (UID: \"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec\") " pod="openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr" Jul 07 15:04:38.142465 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:38.142288 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cpzp\" (UniqueName: \"kubernetes.io/projected/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-kube-api-access-4cpzp\") pod \"e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr\" (UID: \"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec\") " pod="openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr" Jul 07 15:04:38.142465 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:38.142327 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-istio-envoy\") pod \"e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr\" (UID: \"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec\") " pod="openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr" Jul 07 15:04:38.142465 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:38.142352 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-workload-socket\") pod \"e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr\" (UID: \"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec\") " pod="openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr" Jul 07 15:04:38.142465 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:38.142396 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-istio-data\") pod \"e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr\" (UID: \"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec\") " pod="openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr" Jul 07 15:04:38.142465 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:38.142449 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-workload-certs\") pod \"e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr\" (UID: \"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec\") " pod="openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr" Jul 07 15:04:38.142626 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:38.142485 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-istio-token\") pod \"e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr\" (UID: \"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec\") " pod="openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr" Jul 07 15:04:38.142626 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:38.142505 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-istiod-ca-cert\") pod \"e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr\" (UID: \"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec\") " pod="openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr" Jul 07 15:04:38.243336 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:38.243291 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cpzp\" (UniqueName: \"kubernetes.io/projected/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-kube-api-access-4cpzp\") pod \"e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr\" (UID: \"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec\") " pod="openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr" Jul 07 15:04:38.243518 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:38.243349 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-istio-envoy\") pod \"e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr\" (UID: \"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec\") " pod="openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr" Jul 07 15:04:38.243518 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:38.243381 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-workload-socket\") pod \"e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr\" (UID: \"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec\") " pod="openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr" Jul 07 15:04:38.243518 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:38.243434 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-istio-data\") pod \"e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr\" (UID: \"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec\") " pod="openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr" Jul 07 15:04:38.243518 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:38.243484 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-workload-certs\") pod \"e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr\" (UID: \"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec\") " pod="openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr" Jul 07 15:04:38.243790 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:38.243533 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-istio-token\") pod \"e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr\" (UID: \"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec\") " pod="openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr" Jul 07 15:04:38.243790 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:38.243564 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-istiod-ca-cert\") pod \"e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr\" (UID: \"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec\") " pod="openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr" Jul 07 15:04:38.243790 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:38.243617 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-credential-socket\") pod \"e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr\" (UID: \"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec\") " pod="openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr" Jul 07 15:04:38.243790 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:38.243643 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-istio-podinfo\") pod \"e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr\" (UID: \"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec\") " pod="openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr" Jul 07 15:04:38.244437 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:38.244327 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-workload-socket\") pod \"e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr\" (UID: \"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec\") " pod="openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr" Jul 07 15:04:38.244559 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:38.244506 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-workload-certs\") pod \"e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr\" (UID: \"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec\") " pod="openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr" Jul 07 15:04:38.244851 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:38.244798 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-istio-data\") pod \"e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr\" (UID: \"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec\") " pod="openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr" Jul 07 15:04:38.245082 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:38.245063 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-credential-socket\") pod \"e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr\" (UID: \"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec\") " pod="openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr" Jul 07 15:04:38.245408 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:38.245381 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-istiod-ca-cert\") pod \"e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr\" (UID: \"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec\") " pod="openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr" Jul 07 15:04:38.248128 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:38.248041 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-istio-envoy\") pod \"e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr\" (UID: \"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec\") " pod="openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr" Jul 07 15:04:38.248991 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:38.248942 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-istio-podinfo\") pod \"e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr\" (UID: \"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec\") " pod="openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr" Jul 07 15:04:38.252276 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:38.252255 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-istio-token\") pod \"e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr\" (UID: \"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec\") " pod="openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr" Jul 07 15:04:38.253130 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:38.253109 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cpzp\" (UniqueName: \"kubernetes.io/projected/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-kube-api-access-4cpzp\") pod \"e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr\" (UID: \"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec\") " pod="openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr" Jul 07 15:04:38.353619 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:38.353585 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr" Jul 07 15:04:38.498495 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:38.498467 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr"] Jul 07 15:04:38.500819 ip-10-0-136-19 kubenswrapper[2555]: W0707 15:04:38.500789 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a74666c_eaa7_4e2a_a1cb_e4f1a442a9ec.slice/crio-dcce84cc9de7957ef96d86bdd630ba1561e5f954a43f69dff35da9ced9465ef1 WatchSource:0}: Error finding container dcce84cc9de7957ef96d86bdd630ba1561e5f954a43f69dff35da9ced9465ef1: Status 404 returned error can't find the container with id dcce84cc9de7957ef96d86bdd630ba1561e5f954a43f69dff35da9ced9465ef1 Jul 07 15:04:38.503114 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:38.503078 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:04:38.503240 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:38.503148 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:04:38.503240 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:38.503175 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:04:38.637203 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:38.637102 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr" event={"ID":"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec","Type":"ContainerStarted","Data":"b38ff5447abeb21429abfa4ea68c9ba623156c0e070b6209b9b964f2cd229880"} Jul 07 15:04:38.637203 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:38.637168 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr" event={"ID":"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec","Type":"ContainerStarted","Data":"dcce84cc9de7957ef96d86bdd630ba1561e5f954a43f69dff35da9ced9465ef1"} Jul 07 15:04:38.658271 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:38.658196 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr" podStartSLOduration=0.658119787 podStartE2EDuration="658.119787ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:04:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:04:38.657573635 +0000 UTC m=+1820.081760111" watchObservedRunningTime="2026-07-07 15:04:38.658119787 +0000 UTC m=+1820.082306263" Jul 07 15:04:39.352420 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:39.352386 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-e5b77db4-5c9849797d-kzbwx"] Jul 07 15:04:39.355996 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:39.355966 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr" Jul 07 15:04:39.356166 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:39.356079 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-e5b77db4-5c9849797d-kzbwx" Jul 07 15:04:39.359085 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:39.359056 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-e5b77db4-serving-cert\"" Jul 07 15:04:39.359443 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:39.359422 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr" Jul 07 15:04:39.369093 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:39.369064 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-e5b77db4-5c9849797d-kzbwx"] Jul 07 15:04:39.453663 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:39.453633 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22d4529c-7299-4a4e-845f-1b7040b5d653-maas-api-tls\") pod \"maas-api-e2e-ait-e5b77db4-5c9849797d-kzbwx\" (UID: \"22d4529c-7299-4a4e-845f-1b7040b5d653\") " pod="opendatahub/maas-api-e2e-ait-e5b77db4-5c9849797d-kzbwx" Jul 07 15:04:39.453871 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:39.453678 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22d4529c-7299-4a4e-845f-1b7040b5d653-openshift-service-ca\") pod \"maas-api-e2e-ait-e5b77db4-5c9849797d-kzbwx\" (UID: \"22d4529c-7299-4a4e-845f-1b7040b5d653\") " pod="opendatahub/maas-api-e2e-ait-e5b77db4-5c9849797d-kzbwx" Jul 07 15:04:39.453871 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:39.453699 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqbs8\" (UniqueName: \"kubernetes.io/projected/22d4529c-7299-4a4e-845f-1b7040b5d653-kube-api-access-wqbs8\") pod \"maas-api-e2e-ait-e5b77db4-5c9849797d-kzbwx\" (UID: \"22d4529c-7299-4a4e-845f-1b7040b5d653\") " pod="opendatahub/maas-api-e2e-ait-e5b77db4-5c9849797d-kzbwx" Jul 07 15:04:39.554493 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:39.554456 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22d4529c-7299-4a4e-845f-1b7040b5d653-openshift-service-ca\") pod \"maas-api-e2e-ait-e5b77db4-5c9849797d-kzbwx\" (UID: \"22d4529c-7299-4a4e-845f-1b7040b5d653\") " pod="opendatahub/maas-api-e2e-ait-e5b77db4-5c9849797d-kzbwx" Jul 07 15:04:39.554493 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:39.554491 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqbs8\" (UniqueName: \"kubernetes.io/projected/22d4529c-7299-4a4e-845f-1b7040b5d653-kube-api-access-wqbs8\") pod \"maas-api-e2e-ait-e5b77db4-5c9849797d-kzbwx\" (UID: \"22d4529c-7299-4a4e-845f-1b7040b5d653\") " pod="opendatahub/maas-api-e2e-ait-e5b77db4-5c9849797d-kzbwx" Jul 07 15:04:39.554734 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:39.554567 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22d4529c-7299-4a4e-845f-1b7040b5d653-maas-api-tls\") pod \"maas-api-e2e-ait-e5b77db4-5c9849797d-kzbwx\" (UID: \"22d4529c-7299-4a4e-845f-1b7040b5d653\") " pod="opendatahub/maas-api-e2e-ait-e5b77db4-5c9849797d-kzbwx" Jul 07 15:04:39.554734 ip-10-0-136-19 kubenswrapper[2555]: E0707 15:04:39.554660 2555 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-e5b77db4-serving-cert: secret "maas-api-e2e-ait-e5b77db4-serving-cert" not found Jul 07 15:04:39.554734 ip-10-0-136-19 kubenswrapper[2555]: E0707 15:04:39.554716 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22d4529c-7299-4a4e-845f-1b7040b5d653-maas-api-tls podName:22d4529c-7299-4a4e-845f-1b7040b5d653 nodeName:}" failed. No retries permitted until 2026-07-07 15:04:40.054700461 +0000 UTC m=+1821.478886913 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/22d4529c-7299-4a4e-845f-1b7040b5d653-maas-api-tls") pod "maas-api-e2e-ait-e5b77db4-5c9849797d-kzbwx" (UID: "22d4529c-7299-4a4e-845f-1b7040b5d653") : secret "maas-api-e2e-ait-e5b77db4-serving-cert" not found Jul 07 15:04:39.555149 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:39.555129 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22d4529c-7299-4a4e-845f-1b7040b5d653-openshift-service-ca\") pod \"maas-api-e2e-ait-e5b77db4-5c9849797d-kzbwx\" (UID: \"22d4529c-7299-4a4e-845f-1b7040b5d653\") " pod="opendatahub/maas-api-e2e-ait-e5b77db4-5c9849797d-kzbwx" Jul 07 15:04:39.564289 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:39.564266 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqbs8\" (UniqueName: \"kubernetes.io/projected/22d4529c-7299-4a4e-845f-1b7040b5d653-kube-api-access-wqbs8\") pod \"maas-api-e2e-ait-e5b77db4-5c9849797d-kzbwx\" (UID: \"22d4529c-7299-4a4e-845f-1b7040b5d653\") " pod="opendatahub/maas-api-e2e-ait-e5b77db4-5c9849797d-kzbwx" Jul 07 15:04:39.643184 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:39.643101 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr" Jul 07 15:04:39.644074 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:39.644053 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr" Jul 07 15:04:40.060204 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:40.060108 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22d4529c-7299-4a4e-845f-1b7040b5d653-maas-api-tls\") pod \"maas-api-e2e-ait-e5b77db4-5c9849797d-kzbwx\" (UID: \"22d4529c-7299-4a4e-845f-1b7040b5d653\") " pod="opendatahub/maas-api-e2e-ait-e5b77db4-5c9849797d-kzbwx" Jul 07 15:04:40.062623 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:40.062594 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22d4529c-7299-4a4e-845f-1b7040b5d653-maas-api-tls\") pod \"maas-api-e2e-ait-e5b77db4-5c9849797d-kzbwx\" (UID: \"22d4529c-7299-4a4e-845f-1b7040b5d653\") " pod="opendatahub/maas-api-e2e-ait-e5b77db4-5c9849797d-kzbwx" Jul 07 15:04:40.269879 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:40.269840 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-e5b77db4-5c9849797d-kzbwx" Jul 07 15:04:40.418738 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:40.418594 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-e5b77db4-5c9849797d-kzbwx"] Jul 07 15:04:40.430741 ip-10-0-136-19 kubenswrapper[2555]: W0707 15:04:40.430701 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22d4529c_7299_4a4e_845f_1b7040b5d653.slice/crio-07bea1a18b39fcb50fa94db3dd118a2522428607b73fb9264dae9e9ad07b0a0a WatchSource:0}: Error finding container 07bea1a18b39fcb50fa94db3dd118a2522428607b73fb9264dae9e9ad07b0a0a: Status 404 returned error can't find the container with id 07bea1a18b39fcb50fa94db3dd118a2522428607b73fb9264dae9e9ad07b0a0a Jul 07 15:04:40.649469 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:40.649435 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-e5b77db4-5c9849797d-kzbwx" event={"ID":"22d4529c-7299-4a4e-845f-1b7040b5d653","Type":"ContainerStarted","Data":"07bea1a18b39fcb50fa94db3dd118a2522428607b73fb9264dae9e9ad07b0a0a"} Jul 07 15:04:41.656191 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:41.656151 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-e5b77db4-5c9849797d-kzbwx" event={"ID":"22d4529c-7299-4a4e-845f-1b7040b5d653","Type":"ContainerStarted","Data":"64f978b9f851b0be65b11916f90ffe3f316607094282d1d6a293cf0b128a4161"} Jul 07 15:04:41.656666 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:41.656408 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-e5b77db4-5c9849797d-kzbwx" Jul 07 15:04:41.679464 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:41.679408 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-e5b77db4-5c9849797d-kzbwx" podStartSLOduration=2.205544788 podStartE2EDuration="2.67939019s" totalImagesPullingTime="473.845402ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:04:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:04:41.677492678 +0000 UTC m=+1823.101679155" watchObservedRunningTime="2026-07-07 15:04:41.67939019 +0000 UTC m=+1823.103576664" Jul 07 15:04:44.790908 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:44.790871 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-e5b77db4-5c9849797d-kzbwx"] Jul 07 15:04:44.791375 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:44.791075 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-e5b77db4-5c9849797d-kzbwx" podUID="22d4529c-7299-4a4e-845f-1b7040b5d653" containerName="maas-api" containerID="cri-o://64f978b9f851b0be65b11916f90ffe3f316607094282d1d6a293cf0b128a4161" gracePeriod=30 Jul 07 15:04:45.051866 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:45.051790 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-e5b77db4-5c9849797d-kzbwx" Jul 07 15:04:45.109114 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:45.109073 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22d4529c-7299-4a4e-845f-1b7040b5d653-maas-api-tls\") pod \"22d4529c-7299-4a4e-845f-1b7040b5d653\" (UID: \"22d4529c-7299-4a4e-845f-1b7040b5d653\") " Jul 07 15:04:45.109320 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:45.109138 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqbs8\" (UniqueName: \"kubernetes.io/projected/22d4529c-7299-4a4e-845f-1b7040b5d653-kube-api-access-wqbs8\") pod \"22d4529c-7299-4a4e-845f-1b7040b5d653\" (UID: \"22d4529c-7299-4a4e-845f-1b7040b5d653\") " Jul 07 15:04:45.109320 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:45.109164 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22d4529c-7299-4a4e-845f-1b7040b5d653-openshift-service-ca\") pod \"22d4529c-7299-4a4e-845f-1b7040b5d653\" (UID: \"22d4529c-7299-4a4e-845f-1b7040b5d653\") " Jul 07 15:04:45.109599 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:45.109572 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22d4529c-7299-4a4e-845f-1b7040b5d653-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "22d4529c-7299-4a4e-845f-1b7040b5d653" (UID: "22d4529c-7299-4a4e-845f-1b7040b5d653"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:04:45.111311 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:45.111287 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22d4529c-7299-4a4e-845f-1b7040b5d653-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "22d4529c-7299-4a4e-845f-1b7040b5d653" (UID: "22d4529c-7299-4a4e-845f-1b7040b5d653"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:04:45.111397 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:45.111364 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22d4529c-7299-4a4e-845f-1b7040b5d653-kube-api-access-wqbs8" (OuterVolumeSpecName: "kube-api-access-wqbs8") pod "22d4529c-7299-4a4e-845f-1b7040b5d653" (UID: "22d4529c-7299-4a4e-845f-1b7040b5d653"). InnerVolumeSpecName "kube-api-access-wqbs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:04:45.209880 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:45.209846 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22d4529c-7299-4a4e-845f-1b7040b5d653-maas-api-tls\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:04:45.209880 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:45.209876 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqbs8\" (UniqueName: \"kubernetes.io/projected/22d4529c-7299-4a4e-845f-1b7040b5d653-kube-api-access-wqbs8\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:04:45.209880 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:45.209888 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22d4529c-7299-4a4e-845f-1b7040b5d653-openshift-service-ca\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:04:45.676330 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:45.676292 2555 generic.go:356] "Generic (PLEG): container finished" podID="22d4529c-7299-4a4e-845f-1b7040b5d653" containerID="64f978b9f851b0be65b11916f90ffe3f316607094282d1d6a293cf0b128a4161" exitCode=0 Jul 07 15:04:45.676576 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:45.676340 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-e5b77db4-5c9849797d-kzbwx" Jul 07 15:04:45.676576 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:45.676347 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-e5b77db4-5c9849797d-kzbwx" event={"ID":"22d4529c-7299-4a4e-845f-1b7040b5d653","Type":"ContainerDied","Data":"64f978b9f851b0be65b11916f90ffe3f316607094282d1d6a293cf0b128a4161"} Jul 07 15:04:45.676576 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:45.676389 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-e5b77db4-5c9849797d-kzbwx" event={"ID":"22d4529c-7299-4a4e-845f-1b7040b5d653","Type":"ContainerDied","Data":"07bea1a18b39fcb50fa94db3dd118a2522428607b73fb9264dae9e9ad07b0a0a"} Jul 07 15:04:45.676576 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:45.676410 2555 scope.go:117] "RemoveContainer" containerID="64f978b9f851b0be65b11916f90ffe3f316607094282d1d6a293cf0b128a4161" Jul 07 15:04:45.685833 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:45.685813 2555 scope.go:117] "RemoveContainer" containerID="64f978b9f851b0be65b11916f90ffe3f316607094282d1d6a293cf0b128a4161" Jul 07 15:04:45.686150 ip-10-0-136-19 kubenswrapper[2555]: E0707 15:04:45.686131 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64f978b9f851b0be65b11916f90ffe3f316607094282d1d6a293cf0b128a4161\": container with ID starting with 64f978b9f851b0be65b11916f90ffe3f316607094282d1d6a293cf0b128a4161 not found: ID does not exist" containerID="64f978b9f851b0be65b11916f90ffe3f316607094282d1d6a293cf0b128a4161" Jul 07 15:04:45.686222 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:45.686159 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64f978b9f851b0be65b11916f90ffe3f316607094282d1d6a293cf0b128a4161"} err="failed to get container status \"64f978b9f851b0be65b11916f90ffe3f316607094282d1d6a293cf0b128a4161\": rpc error: code = NotFound desc = could not find container \"64f978b9f851b0be65b11916f90ffe3f316607094282d1d6a293cf0b128a4161\": container with ID starting with 64f978b9f851b0be65b11916f90ffe3f316607094282d1d6a293cf0b128a4161 not found: ID does not exist" Jul 07 15:04:45.699491 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:45.699419 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-e5b77db4-5c9849797d-kzbwx"] Jul 07 15:04:45.704266 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:45.704235 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-e5b77db4-5c9849797d-kzbwx"] Jul 07 15:04:47.042658 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:47.042619 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22d4529c-7299-4a4e-845f-1b7040b5d653" path="/var/lib/kubelet/pods/22d4529c-7299-4a4e-845f-1b7040b5d653/volumes" Jul 07 15:04:50.480036 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:50.479997 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr"] Jul 07 15:04:50.480548 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:50.480197 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr" podUID="3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec" containerName="istio-proxy" containerID="cri-o://b38ff5447abeb21429abfa4ea68c9ba623156c0e070b6209b9b964f2cd229880" gracePeriod=30 Jul 07 15:04:54.645363 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:54.645327 2555 patch_prober.go:28] interesting pod/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Jul 07 15:04:54.645779 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:54.645387 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr" podUID="3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 15:04:55.724416 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:55.724384 2555 generic.go:356] "Generic (PLEG): container finished" podID="3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec" containerID="b38ff5447abeb21429abfa4ea68c9ba623156c0e070b6209b9b964f2cd229880" exitCode=0 Jul 07 15:04:55.724851 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:55.724463 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr" event={"ID":"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec","Type":"ContainerDied","Data":"b38ff5447abeb21429abfa4ea68c9ba623156c0e070b6209b9b964f2cd229880"} Jul 07 15:04:55.724851 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:55.724518 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr" event={"ID":"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec","Type":"ContainerDied","Data":"dcce84cc9de7957ef96d86bdd630ba1561e5f954a43f69dff35da9ced9465ef1"} Jul 07 15:04:55.724851 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:55.724536 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcce84cc9de7957ef96d86bdd630ba1561e5f954a43f69dff35da9ced9465ef1" Jul 07 15:04:55.732255 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:55.732227 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr" Jul 07 15:04:55.912568 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:55.912521 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-istio-data\") pod \"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec\" (UID: \"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec\") " Jul 07 15:04:55.912568 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:55.912570 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-credential-socket\") pod \"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec\" (UID: \"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec\") " Jul 07 15:04:55.912872 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:55.912597 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-istio-podinfo\") pod \"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec\" (UID: \"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec\") " Jul 07 15:04:55.912872 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:55.912620 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-workload-certs\") pod \"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec\" (UID: \"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec\") " Jul 07 15:04:55.912872 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:55.912646 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-workload-socket\") pod \"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec\" (UID: \"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec\") " Jul 07 15:04:55.912872 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:55.912805 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-istiod-ca-cert\") pod \"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec\" (UID: \"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec\") " Jul 07 15:04:55.913083 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:55.912898 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4cpzp\" (UniqueName: \"kubernetes.io/projected/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-kube-api-access-4cpzp\") pod \"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec\" (UID: \"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec\") " Jul 07 15:04:55.913083 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:55.912927 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-istio-token\") pod \"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec\" (UID: \"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec\") " Jul 07 15:04:55.913083 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:55.912934 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec" (UID: "3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:04:55.913083 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:55.912915 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-istio-data" (OuterVolumeSpecName: "istio-data") pod "3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec" (UID: "3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:04:55.913083 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:55.912963 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec" (UID: "3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:04:55.913083 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:55.913021 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-istio-envoy\") pod \"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec\" (UID: \"3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec\") " Jul 07 15:04:55.913410 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:55.913102 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec" (UID: "3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:04:55.913410 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:55.913236 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec" (UID: "3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:04:55.913410 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:55.913365 2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-istio-data\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:04:55.913410 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:55.913385 2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-credential-socket\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:04:55.913410 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:55.913399 2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-workload-certs\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:04:55.913602 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:55.913413 2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-workload-socket\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:04:55.913602 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:55.913426 2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-istiod-ca-cert\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:04:55.915113 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:55.915089 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec" (UID: "3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 15:04:55.915215 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:55.915131 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-kube-api-access-4cpzp" (OuterVolumeSpecName: "kube-api-access-4cpzp") pod "3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec" (UID: "3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec"). InnerVolumeSpecName "kube-api-access-4cpzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:04:55.915399 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:55.915380 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-istio-token" (OuterVolumeSpecName: "istio-token") pod "3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec" (UID: "3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:04:55.915399 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:55.915387 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec" (UID: "3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:04:56.014199 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:56.014156 2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-istio-podinfo\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:04:56.014199 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:56.014191 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cpzp\" (UniqueName: \"kubernetes.io/projected/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-kube-api-access-4cpzp\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:04:56.014199 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:56.014205 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-istio-token\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:04:56.014468 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:56.014218 2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec-istio-envoy\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:04:56.729863 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:56.729820 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr" Jul 07 15:04:56.752891 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:56.752852 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr"] Jul 07 15:04:56.757356 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:56.757328 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e5b77db4-openshift-default-5954fcd5d7-x26pr"] Jul 07 15:04:56.903841 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:56.903801 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv"] Jul 07 15:04:56.904230 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:56.904213 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec" containerName="istio-proxy" Jul 07 15:04:56.904316 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:56.904233 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec" containerName="istio-proxy" Jul 07 15:04:56.904316 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:56.904269 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22d4529c-7299-4a4e-845f-1b7040b5d653" containerName="maas-api" Jul 07 15:04:56.904316 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:56.904280 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="22d4529c-7299-4a4e-845f-1b7040b5d653" containerName="maas-api" Jul 07 15:04:56.904474 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:56.904363 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec" containerName="istio-proxy" Jul 07 15:04:56.904474 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:56.904377 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="22d4529c-7299-4a4e-845f-1b7040b5d653" containerName="maas-api" Jul 07 15:04:56.907251 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:56.907226 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv" Jul 07 15:04:56.909967 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:56.909945 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-30c46ab0-openshift-default-dockercfg-d79jm\"" Jul 07 15:04:56.921299 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:56.921276 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv"] Jul 07 15:04:57.023328 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:57.023229 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-istiod-ca-cert\") pod \"e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv\" (UID: \"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649\") " pod="openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv" Jul 07 15:04:57.023328 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:57.023267 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7kdb\" (UniqueName: \"kubernetes.io/projected/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-kube-api-access-q7kdb\") pod \"e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv\" (UID: \"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649\") " pod="openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv" Jul 07 15:04:57.023328 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:57.023293 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-istio-token\") pod \"e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv\" (UID: \"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649\") " pod="openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv" Jul 07 15:04:57.023328 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:57.023314 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-istio-podinfo\") pod \"e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv\" (UID: \"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649\") " pod="openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv" Jul 07 15:04:57.023701 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:57.023429 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-workload-certs\") pod \"e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv\" (UID: \"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649\") " pod="openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv" Jul 07 15:04:57.023701 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:57.023481 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-istio-envoy\") pod \"e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv\" (UID: \"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649\") " pod="openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv" Jul 07 15:04:57.023701 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:57.023520 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-credential-socket\") pod \"e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv\" (UID: \"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649\") " pod="openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv" Jul 07 15:04:57.023701 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:57.023616 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-workload-socket\") pod \"e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv\" (UID: \"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649\") " pod="openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv" Jul 07 15:04:57.023701 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:57.023659 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-istio-data\") pod \"e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv\" (UID: \"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649\") " pod="openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv" Jul 07 15:04:57.042407 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:57.042375 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec" path="/var/lib/kubelet/pods/3a74666c-eaa7-4e2a-a1cb-e4f1a442a9ec/volumes" Jul 07 15:04:57.124846 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:57.124806 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-istio-envoy\") pod \"e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv\" (UID: \"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649\") " pod="openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv" Jul 07 15:04:57.125057 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:57.124854 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-credential-socket\") pod \"e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv\" (UID: \"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649\") " pod="openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv" Jul 07 15:04:57.125057 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:57.124902 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-workload-socket\") pod \"e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv\" (UID: \"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649\") " pod="openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv" Jul 07 15:04:57.125057 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:57.124928 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-istio-data\") pod \"e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv\" (UID: \"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649\") " pod="openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv" Jul 07 15:04:57.125057 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:57.124946 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-istiod-ca-cert\") pod \"e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv\" (UID: \"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649\") " pod="openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv" Jul 07 15:04:57.125057 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:57.124964 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7kdb\" (UniqueName: \"kubernetes.io/projected/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-kube-api-access-q7kdb\") pod \"e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv\" (UID: \"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649\") " pod="openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv" Jul 07 15:04:57.125057 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:57.124996 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-istio-token\") pod \"e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv\" (UID: \"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649\") " pod="openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv" Jul 07 15:04:57.125057 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:57.125024 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-istio-podinfo\") pod \"e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv\" (UID: \"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649\") " pod="openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv" Jul 07 15:04:57.125427 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:57.125081 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-workload-certs\") pod \"e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv\" (UID: \"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649\") " pod="openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv" Jul 07 15:04:57.125427 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:57.125395 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-credential-socket\") pod \"e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv\" (UID: \"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649\") " pod="openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv" Jul 07 15:04:57.125530 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:57.125456 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-workload-socket\") pod \"e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv\" (UID: \"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649\") " pod="openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv" Jul 07 15:04:57.125530 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:57.125493 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-istio-data\") pod \"e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv\" (UID: \"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649\") " pod="openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv" Jul 07 15:04:57.125637 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:57.125558 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-workload-certs\") pod \"e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv\" (UID: \"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649\") " pod="openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv" Jul 07 15:04:57.125814 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:57.125745 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-istiod-ca-cert\") pod \"e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv\" (UID: \"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649\") " pod="openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv" Jul 07 15:04:57.127275 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:57.127253 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-istio-envoy\") pod \"e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv\" (UID: \"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649\") " pod="openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv" Jul 07 15:04:57.127635 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:57.127614 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-istio-podinfo\") pod \"e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv\" (UID: \"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649\") " pod="openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv" Jul 07 15:04:57.134830 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:57.134800 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-istio-token\") pod \"e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv\" (UID: \"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649\") " pod="openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv" Jul 07 15:04:57.135056 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:57.135035 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7kdb\" (UniqueName: \"kubernetes.io/projected/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-kube-api-access-q7kdb\") pod \"e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv\" (UID: \"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649\") " pod="openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv" Jul 07 15:04:57.219680 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:57.219637 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv" Jul 07 15:04:57.362973 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:57.362797 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv"] Jul 07 15:04:57.365702 ip-10-0-136-19 kubenswrapper[2555]: W0707 15:04:57.365676 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod374bb3e2_9a42_4d24_9c81_0d1b2f9bf649.slice/crio-53efad5e8de9fb1ad6be6e59b2f8670b3a7534fb1416c000c54d4784bdff8850 WatchSource:0}: Error finding container 53efad5e8de9fb1ad6be6e59b2f8670b3a7534fb1416c000c54d4784bdff8850: Status 404 returned error can't find the container with id 53efad5e8de9fb1ad6be6e59b2f8670b3a7534fb1416c000c54d4784bdff8850 Jul 07 15:04:57.367875 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:57.367841 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:04:57.367968 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:57.367907 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:04:57.368005 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:57.367971 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:04:57.736261 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:57.736220 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv" event={"ID":"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649","Type":"ContainerStarted","Data":"16f82920097551d047de386a4a93e409bad59b2d32ef2fa56969cab11afd9cd9"} Jul 07 15:04:57.736813 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:57.736281 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv" event={"ID":"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649","Type":"ContainerStarted","Data":"53efad5e8de9fb1ad6be6e59b2f8670b3a7534fb1416c000c54d4784bdff8850"} Jul 07 15:04:57.759774 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:57.759697 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv" podStartSLOduration=1.759677994 podStartE2EDuration="1.759677994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:04:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:04:57.759047237 +0000 UTC m=+1839.183233712" watchObservedRunningTime="2026-07-07 15:04:57.759677994 +0000 UTC m=+1839.183864469" Jul 07 15:04:58.160067 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:58.160032 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-30c46ab0-84758f4876-xsvnh"] Jul 07 15:04:58.163180 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:58.163160 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-30c46ab0-84758f4876-xsvnh" Jul 07 15:04:58.167964 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:58.167935 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-30c46ab0-serving-cert\"" Jul 07 15:04:58.181092 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:58.181068 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-30c46ab0-84758f4876-xsvnh"] Jul 07 15:04:58.220540 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:58.220505 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv" Jul 07 15:04:58.225667 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:58.225642 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv" Jul 07 15:04:58.337088 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:58.337054 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93ca9fad-cc95-49dd-b05a-11315a79069e-maas-api-tls\") pod \"maas-api-e2e-derive-30c46ab0-84758f4876-xsvnh\" (UID: \"93ca9fad-cc95-49dd-b05a-11315a79069e\") " pod="opendatahub/maas-api-e2e-derive-30c46ab0-84758f4876-xsvnh" Jul 07 15:04:58.337299 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:58.337095 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vz4z\" (UniqueName: \"kubernetes.io/projected/93ca9fad-cc95-49dd-b05a-11315a79069e-kube-api-access-5vz4z\") pod \"maas-api-e2e-derive-30c46ab0-84758f4876-xsvnh\" (UID: \"93ca9fad-cc95-49dd-b05a-11315a79069e\") " pod="opendatahub/maas-api-e2e-derive-30c46ab0-84758f4876-xsvnh" Jul 07 15:04:58.337299 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:58.337254 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93ca9fad-cc95-49dd-b05a-11315a79069e-openshift-service-ca\") pod \"maas-api-e2e-derive-30c46ab0-84758f4876-xsvnh\" (UID: \"93ca9fad-cc95-49dd-b05a-11315a79069e\") " pod="opendatahub/maas-api-e2e-derive-30c46ab0-84758f4876-xsvnh" Jul 07 15:04:58.438614 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:58.438520 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93ca9fad-cc95-49dd-b05a-11315a79069e-openshift-service-ca\") pod \"maas-api-e2e-derive-30c46ab0-84758f4876-xsvnh\" (UID: \"93ca9fad-cc95-49dd-b05a-11315a79069e\") " pod="opendatahub/maas-api-e2e-derive-30c46ab0-84758f4876-xsvnh" Jul 07 15:04:58.438614 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:58.438578 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93ca9fad-cc95-49dd-b05a-11315a79069e-maas-api-tls\") pod \"maas-api-e2e-derive-30c46ab0-84758f4876-xsvnh\" (UID: \"93ca9fad-cc95-49dd-b05a-11315a79069e\") " pod="opendatahub/maas-api-e2e-derive-30c46ab0-84758f4876-xsvnh" Jul 07 15:04:58.438614 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:58.438604 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vz4z\" (UniqueName: \"kubernetes.io/projected/93ca9fad-cc95-49dd-b05a-11315a79069e-kube-api-access-5vz4z\") pod \"maas-api-e2e-derive-30c46ab0-84758f4876-xsvnh\" (UID: \"93ca9fad-cc95-49dd-b05a-11315a79069e\") " pod="opendatahub/maas-api-e2e-derive-30c46ab0-84758f4876-xsvnh" Jul 07 15:04:58.438886 ip-10-0-136-19 kubenswrapper[2555]: E0707 15:04:58.438798 2555 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-derive-30c46ab0-serving-cert: secret "maas-api-e2e-derive-30c46ab0-serving-cert" not found Jul 07 15:04:58.438930 ip-10-0-136-19 kubenswrapper[2555]: E0707 15:04:58.438886 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93ca9fad-cc95-49dd-b05a-11315a79069e-maas-api-tls podName:93ca9fad-cc95-49dd-b05a-11315a79069e nodeName:}" failed. No retries permitted until 2026-07-07 15:04:58.938864239 +0000 UTC m=+1840.363050715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/93ca9fad-cc95-49dd-b05a-11315a79069e-maas-api-tls") pod "maas-api-e2e-derive-30c46ab0-84758f4876-xsvnh" (UID: "93ca9fad-cc95-49dd-b05a-11315a79069e") : secret "maas-api-e2e-derive-30c46ab0-serving-cert" not found Jul 07 15:04:58.439228 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:58.439209 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93ca9fad-cc95-49dd-b05a-11315a79069e-openshift-service-ca\") pod \"maas-api-e2e-derive-30c46ab0-84758f4876-xsvnh\" (UID: \"93ca9fad-cc95-49dd-b05a-11315a79069e\") " pod="opendatahub/maas-api-e2e-derive-30c46ab0-84758f4876-xsvnh" Jul 07 15:04:58.451851 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:58.451819 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vz4z\" (UniqueName: \"kubernetes.io/projected/93ca9fad-cc95-49dd-b05a-11315a79069e-kube-api-access-5vz4z\") pod \"maas-api-e2e-derive-30c46ab0-84758f4876-xsvnh\" (UID: \"93ca9fad-cc95-49dd-b05a-11315a79069e\") " pod="opendatahub/maas-api-e2e-derive-30c46ab0-84758f4876-xsvnh" Jul 07 15:04:58.741285 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:58.741186 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv" Jul 07 15:04:58.742125 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:58.742108 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv" Jul 07 15:04:58.943510 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:58.943467 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93ca9fad-cc95-49dd-b05a-11315a79069e-maas-api-tls\") pod \"maas-api-e2e-derive-30c46ab0-84758f4876-xsvnh\" (UID: \"93ca9fad-cc95-49dd-b05a-11315a79069e\") " pod="opendatahub/maas-api-e2e-derive-30c46ab0-84758f4876-xsvnh" Jul 07 15:04:58.946090 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:58.946054 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93ca9fad-cc95-49dd-b05a-11315a79069e-maas-api-tls\") pod \"maas-api-e2e-derive-30c46ab0-84758f4876-xsvnh\" (UID: \"93ca9fad-cc95-49dd-b05a-11315a79069e\") " pod="opendatahub/maas-api-e2e-derive-30c46ab0-84758f4876-xsvnh" Jul 07 15:04:59.074618 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:59.074514 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-30c46ab0-84758f4876-xsvnh" Jul 07 15:04:59.218894 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:59.218866 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-30c46ab0-84758f4876-xsvnh"] Jul 07 15:04:59.221598 ip-10-0-136-19 kubenswrapper[2555]: W0707 15:04:59.221570 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93ca9fad_cc95_49dd_b05a_11315a79069e.slice/crio-340ff80aaa34629f34471a1294e2d0b11379afeb9792740627766c5e919fee4a WatchSource:0}: Error finding container 340ff80aaa34629f34471a1294e2d0b11379afeb9792740627766c5e919fee4a: Status 404 returned error can't find the container with id 340ff80aaa34629f34471a1294e2d0b11379afeb9792740627766c5e919fee4a Jul 07 15:04:59.748669 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:04:59.748632 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-30c46ab0-84758f4876-xsvnh" event={"ID":"93ca9fad-cc95-49dd-b05a-11315a79069e","Type":"ContainerStarted","Data":"340ff80aaa34629f34471a1294e2d0b11379afeb9792740627766c5e919fee4a"} Jul 07 15:05:00.755057 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:00.755011 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-30c46ab0-84758f4876-xsvnh" event={"ID":"93ca9fad-cc95-49dd-b05a-11315a79069e","Type":"ContainerStarted","Data":"71d6a76dbc29a7722cd56c06b77498c55163a3276436fee34de3ef6cb5053a45"} Jul 07 15:05:00.755544 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:00.755297 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-30c46ab0-84758f4876-xsvnh" Jul 07 15:05:00.776990 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:00.776934 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-30c46ab0-84758f4876-xsvnh" podStartSLOduration=1.91456559 podStartE2EDuration="2.776917231s" totalImagesPullingTime="862.351641ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:04:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:05:00.775412535 +0000 UTC m=+1842.199599011" watchObservedRunningTime="2026-07-07 15:05:00.776917231 +0000 UTC m=+1842.201103706" Jul 07 15:05:02.691676 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:02.691640 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv"] Jul 07 15:05:02.692103 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:02.691856 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv" podUID="374bb3e2-9a42-4d24-9c81-0d1b2f9bf649" containerName="istio-proxy" containerID="cri-o://16f82920097551d047de386a4a93e409bad59b2d32ef2fa56969cab11afd9cd9" gracePeriod=30 Jul 07 15:05:02.921496 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:02.921453 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-30c46ab0-84758f4876-xsvnh"] Jul 07 15:05:02.921949 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:02.921919 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-30c46ab0-84758f4876-xsvnh" podUID="93ca9fad-cc95-49dd-b05a-11315a79069e" containerName="maas-api" containerID="cri-o://71d6a76dbc29a7722cd56c06b77498c55163a3276436fee34de3ef6cb5053a45" gracePeriod=30 Jul 07 15:05:03.214297 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:03.214271 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-30c46ab0-84758f4876-xsvnh" Jul 07 15:05:03.280122 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:03.280090 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93ca9fad-cc95-49dd-b05a-11315a79069e-maas-api-tls\") pod \"93ca9fad-cc95-49dd-b05a-11315a79069e\" (UID: \"93ca9fad-cc95-49dd-b05a-11315a79069e\") " Jul 07 15:05:03.280122 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:03.280124 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93ca9fad-cc95-49dd-b05a-11315a79069e-openshift-service-ca\") pod \"93ca9fad-cc95-49dd-b05a-11315a79069e\" (UID: \"93ca9fad-cc95-49dd-b05a-11315a79069e\") " Jul 07 15:05:03.280362 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:03.280202 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vz4z\" (UniqueName: \"kubernetes.io/projected/93ca9fad-cc95-49dd-b05a-11315a79069e-kube-api-access-5vz4z\") pod \"93ca9fad-cc95-49dd-b05a-11315a79069e\" (UID: \"93ca9fad-cc95-49dd-b05a-11315a79069e\") " Jul 07 15:05:03.280522 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:03.280496 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93ca9fad-cc95-49dd-b05a-11315a79069e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "93ca9fad-cc95-49dd-b05a-11315a79069e" (UID: "93ca9fad-cc95-49dd-b05a-11315a79069e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:05:03.282279 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:03.282241 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93ca9fad-cc95-49dd-b05a-11315a79069e-kube-api-access-5vz4z" (OuterVolumeSpecName: "kube-api-access-5vz4z") pod "93ca9fad-cc95-49dd-b05a-11315a79069e" (UID: "93ca9fad-cc95-49dd-b05a-11315a79069e"). InnerVolumeSpecName "kube-api-access-5vz4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:05:03.282279 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:03.282255 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93ca9fad-cc95-49dd-b05a-11315a79069e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "93ca9fad-cc95-49dd-b05a-11315a79069e" (UID: "93ca9fad-cc95-49dd-b05a-11315a79069e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:05:03.380852 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:03.380810 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vz4z\" (UniqueName: \"kubernetes.io/projected/93ca9fad-cc95-49dd-b05a-11315a79069e-kube-api-access-5vz4z\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:05:03.380852 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:03.380845 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93ca9fad-cc95-49dd-b05a-11315a79069e-maas-api-tls\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:05:03.380852 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:03.380854 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93ca9fad-cc95-49dd-b05a-11315a79069e-openshift-service-ca\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:05:03.774860 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:03.774744 2555 generic.go:356] "Generic (PLEG): container finished" podID="93ca9fad-cc95-49dd-b05a-11315a79069e" containerID="71d6a76dbc29a7722cd56c06b77498c55163a3276436fee34de3ef6cb5053a45" exitCode=0 Jul 07 15:05:03.774860 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:03.774779 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-30c46ab0-84758f4876-xsvnh" event={"ID":"93ca9fad-cc95-49dd-b05a-11315a79069e","Type":"ContainerDied","Data":"71d6a76dbc29a7722cd56c06b77498c55163a3276436fee34de3ef6cb5053a45"} Jul 07 15:05:03.774860 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:03.774825 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-30c46ab0-84758f4876-xsvnh" Jul 07 15:05:03.775452 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:03.774865 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-30c46ab0-84758f4876-xsvnh" event={"ID":"93ca9fad-cc95-49dd-b05a-11315a79069e","Type":"ContainerDied","Data":"340ff80aaa34629f34471a1294e2d0b11379afeb9792740627766c5e919fee4a"} Jul 07 15:05:03.775452 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:03.774888 2555 scope.go:117] "RemoveContainer" containerID="71d6a76dbc29a7722cd56c06b77498c55163a3276436fee34de3ef6cb5053a45" Jul 07 15:05:03.784543 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:03.784521 2555 scope.go:117] "RemoveContainer" containerID="71d6a76dbc29a7722cd56c06b77498c55163a3276436fee34de3ef6cb5053a45" Jul 07 15:05:03.784898 ip-10-0-136-19 kubenswrapper[2555]: E0707 15:05:03.784878 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71d6a76dbc29a7722cd56c06b77498c55163a3276436fee34de3ef6cb5053a45\": container with ID starting with 71d6a76dbc29a7722cd56c06b77498c55163a3276436fee34de3ef6cb5053a45 not found: ID does not exist" containerID="71d6a76dbc29a7722cd56c06b77498c55163a3276436fee34de3ef6cb5053a45" Jul 07 15:05:03.784956 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:03.784908 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71d6a76dbc29a7722cd56c06b77498c55163a3276436fee34de3ef6cb5053a45"} err="failed to get container status \"71d6a76dbc29a7722cd56c06b77498c55163a3276436fee34de3ef6cb5053a45\": rpc error: code = NotFound desc = could not find container \"71d6a76dbc29a7722cd56c06b77498c55163a3276436fee34de3ef6cb5053a45\": container with ID starting with 71d6a76dbc29a7722cd56c06b77498c55163a3276436fee34de3ef6cb5053a45 not found: ID does not exist" Jul 07 15:05:03.797490 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:03.797457 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-30c46ab0-84758f4876-xsvnh"] Jul 07 15:05:03.801592 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:03.801564 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-30c46ab0-84758f4876-xsvnh"] Jul 07 15:05:05.045905 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:05.045861 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93ca9fad-cc95-49dd-b05a-11315a79069e" path="/var/lib/kubelet/pods/93ca9fad-cc95-49dd-b05a-11315a79069e/volumes" Jul 07 15:05:08.655258 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:08.655229 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv" Jul 07 15:05:08.729225 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:08.729125 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-istio-envoy\") pod \"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649\" (UID: \"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649\") " Jul 07 15:05:08.729225 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:08.729195 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-workload-certs\") pod \"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649\" (UID: \"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649\") " Jul 07 15:05:08.729487 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:08.729236 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-istiod-ca-cert\") pod \"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649\" (UID: \"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649\") " Jul 07 15:05:08.729487 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:08.729252 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-credential-socket\") pod \"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649\" (UID: \"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649\") " Jul 07 15:05:08.729487 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:08.729283 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-istio-data\") pod \"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649\" (UID: \"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649\") " Jul 07 15:05:08.729487 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:08.729310 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7kdb\" (UniqueName: \"kubernetes.io/projected/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-kube-api-access-q7kdb\") pod \"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649\" (UID: \"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649\") " Jul 07 15:05:08.729487 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:08.729331 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-istio-token\") pod \"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649\" (UID: \"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649\") " Jul 07 15:05:08.729487 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:08.729390 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-istio-podinfo\") pod \"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649\" (UID: \"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649\") " Jul 07 15:05:08.729487 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:08.729408 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-workload-socket\") pod \"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649\" (UID: \"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649\") " Jul 07 15:05:08.729900 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:08.729797 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "374bb3e2-9a42-4d24-9c81-0d1b2f9bf649" (UID: "374bb3e2-9a42-4d24-9c81-0d1b2f9bf649"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:05:08.729900 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:08.729861 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "374bb3e2-9a42-4d24-9c81-0d1b2f9bf649" (UID: "374bb3e2-9a42-4d24-9c81-0d1b2f9bf649"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:05:08.730011 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:08.729909 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "374bb3e2-9a42-4d24-9c81-0d1b2f9bf649" (UID: "374bb3e2-9a42-4d24-9c81-0d1b2f9bf649"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:05:08.730011 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:08.729976 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "374bb3e2-9a42-4d24-9c81-0d1b2f9bf649" (UID: "374bb3e2-9a42-4d24-9c81-0d1b2f9bf649"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:05:08.730411 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:08.730382 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-istio-data" (OuterVolumeSpecName: "istio-data") pod "374bb3e2-9a42-4d24-9c81-0d1b2f9bf649" (UID: "374bb3e2-9a42-4d24-9c81-0d1b2f9bf649"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:05:08.731613 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:08.731584 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-istio-token" (OuterVolumeSpecName: "istio-token") pod "374bb3e2-9a42-4d24-9c81-0d1b2f9bf649" (UID: "374bb3e2-9a42-4d24-9c81-0d1b2f9bf649"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:05:08.731613 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:08.731584 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-kube-api-access-q7kdb" (OuterVolumeSpecName: "kube-api-access-q7kdb") pod "374bb3e2-9a42-4d24-9c81-0d1b2f9bf649" (UID: "374bb3e2-9a42-4d24-9c81-0d1b2f9bf649"). InnerVolumeSpecName "kube-api-access-q7kdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:05:08.731613 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:08.731594 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "374bb3e2-9a42-4d24-9c81-0d1b2f9bf649" (UID: "374bb3e2-9a42-4d24-9c81-0d1b2f9bf649"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:05:08.731847 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:08.731829 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "374bb3e2-9a42-4d24-9c81-0d1b2f9bf649" (UID: "374bb3e2-9a42-4d24-9c81-0d1b2f9bf649"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 15:05:08.800695 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:08.800654 2555 generic.go:356] "Generic (PLEG): container finished" podID="374bb3e2-9a42-4d24-9c81-0d1b2f9bf649" containerID="16f82920097551d047de386a4a93e409bad59b2d32ef2fa56969cab11afd9cd9" exitCode=0 Jul 07 15:05:08.800894 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:08.800702 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv" Jul 07 15:05:08.800894 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:08.800743 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv" event={"ID":"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649","Type":"ContainerDied","Data":"16f82920097551d047de386a4a93e409bad59b2d32ef2fa56969cab11afd9cd9"} Jul 07 15:05:08.800894 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:08.800815 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv" event={"ID":"374bb3e2-9a42-4d24-9c81-0d1b2f9bf649","Type":"ContainerDied","Data":"53efad5e8de9fb1ad6be6e59b2f8670b3a7534fb1416c000c54d4784bdff8850"} Jul 07 15:05:08.800894 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:08.800840 2555 scope.go:117] "RemoveContainer" containerID="16f82920097551d047de386a4a93e409bad59b2d32ef2fa56969cab11afd9cd9" Jul 07 15:05:08.810351 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:08.810333 2555 scope.go:117] "RemoveContainer" containerID="16f82920097551d047de386a4a93e409bad59b2d32ef2fa56969cab11afd9cd9" Jul 07 15:05:08.810689 ip-10-0-136-19 kubenswrapper[2555]: E0707 15:05:08.810668 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16f82920097551d047de386a4a93e409bad59b2d32ef2fa56969cab11afd9cd9\": container with ID starting with 16f82920097551d047de386a4a93e409bad59b2d32ef2fa56969cab11afd9cd9 not found: ID does not exist" containerID="16f82920097551d047de386a4a93e409bad59b2d32ef2fa56969cab11afd9cd9" Jul 07 15:05:08.810738 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:08.810700 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16f82920097551d047de386a4a93e409bad59b2d32ef2fa56969cab11afd9cd9"} err="failed to get container status \"16f82920097551d047de386a4a93e409bad59b2d32ef2fa56969cab11afd9cd9\": rpc error: code = NotFound desc = could not find container \"16f82920097551d047de386a4a93e409bad59b2d32ef2fa56969cab11afd9cd9\": container with ID starting with 16f82920097551d047de386a4a93e409bad59b2d32ef2fa56969cab11afd9cd9 not found: ID does not exist" Jul 07 15:05:08.823877 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:08.823844 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv"] Jul 07 15:05:08.828039 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:08.827998 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-30c46ab0-openshift-default-6f669d44f9-hvpxv"] Jul 07 15:05:08.830464 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:08.830442 2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-workload-certs\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:05:08.830464 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:08.830465 2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-istiod-ca-cert\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:05:08.830591 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:08.830474 2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-credential-socket\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:05:08.830591 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:08.830483 2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-istio-data\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:05:08.830591 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:08.830492 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7kdb\" (UniqueName: \"kubernetes.io/projected/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-kube-api-access-q7kdb\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:05:08.830591 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:08.830500 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-istio-token\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:05:08.830591 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:08.830508 2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-istio-podinfo\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:05:08.830591 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:08.830516 2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-workload-socket\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:05:08.830591 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:08.830523 2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649-istio-envoy\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:05:09.042137 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:05:09.042046 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="374bb3e2-9a42-4d24-9c81-0d1b2f9bf649" path="/var/lib/kubelet/pods/374bb3e2-9a42-4d24-9c81-0d1b2f9bf649/volumes" Jul 07 15:06:44.921470 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:44.921432 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj"] Jul 07 15:06:44.921974 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:44.921862 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="374bb3e2-9a42-4d24-9c81-0d1b2f9bf649" containerName="istio-proxy" Jul 07 15:06:44.921974 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:44.921875 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="374bb3e2-9a42-4d24-9c81-0d1b2f9bf649" containerName="istio-proxy" Jul 07 15:06:44.921974 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:44.921896 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93ca9fad-cc95-49dd-b05a-11315a79069e" containerName="maas-api" Jul 07 15:06:44.921974 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:44.921901 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="93ca9fad-cc95-49dd-b05a-11315a79069e" containerName="maas-api" Jul 07 15:06:44.921974 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:44.921954 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="374bb3e2-9a42-4d24-9c81-0d1b2f9bf649" containerName="istio-proxy" Jul 07 15:06:44.921974 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:44.921962 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="93ca9fad-cc95-49dd-b05a-11315a79069e" containerName="maas-api" Jul 07 15:06:44.925044 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:44.925028 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" Jul 07 15:06:44.928203 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:44.928178 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-06869d8e-openshift-default-dockercfg-65kjb\"" Jul 07 15:06:44.946841 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:44.946817 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj"] Jul 07 15:06:44.994614 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:44.994576 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-workload-socket\") pod \"e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj\" (UID: \"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7\") " pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" Jul 07 15:06:44.994614 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:44.994613 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-istiod-ca-cert\") pod \"e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj\" (UID: \"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7\") " pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" Jul 07 15:06:44.994916 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:44.994639 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-workload-certs\") pod \"e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj\" (UID: \"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7\") " pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" Jul 07 15:06:44.994916 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:44.994703 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-credential-socket\") pod \"e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj\" (UID: \"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7\") " pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" Jul 07 15:06:44.994916 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:44.994725 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-istio-token\") pod \"e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj\" (UID: \"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7\") " pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" Jul 07 15:06:44.994916 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:44.994776 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dklt9\" (UniqueName: \"kubernetes.io/projected/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-kube-api-access-dklt9\") pod \"e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj\" (UID: \"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7\") " pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" Jul 07 15:06:44.994916 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:44.994810 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-istio-podinfo\") pod \"e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj\" (UID: \"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7\") " pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" Jul 07 15:06:44.994916 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:44.994846 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-istio-envoy\") pod \"e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj\" (UID: \"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7\") " pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" Jul 07 15:06:44.994916 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:44.994899 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-istio-data\") pod \"e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj\" (UID: \"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7\") " pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" Jul 07 15:06:45.095500 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:45.095455 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-istio-data\") pod \"e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj\" (UID: \"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7\") " pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" Jul 07 15:06:45.095718 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:45.095572 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-workload-socket\") pod \"e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj\" (UID: \"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7\") " pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" Jul 07 15:06:45.095718 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:45.095612 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-istiod-ca-cert\") pod \"e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj\" (UID: \"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7\") " pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" Jul 07 15:06:45.095718 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:45.095644 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-workload-certs\") pod \"e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj\" (UID: \"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7\") " pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" Jul 07 15:06:45.095944 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:45.095786 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-credential-socket\") pod \"e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj\" (UID: \"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7\") " pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" Jul 07 15:06:45.096091 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:45.096066 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-workload-socket\") pod \"e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj\" (UID: \"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7\") " pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" Jul 07 15:06:45.096190 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:45.096155 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-workload-certs\") pod \"e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj\" (UID: \"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7\") " pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" Jul 07 15:06:45.096254 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:45.096190 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-istio-data\") pod \"e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj\" (UID: \"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7\") " pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" Jul 07 15:06:45.096254 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:45.096204 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-istio-token\") pod \"e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj\" (UID: \"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7\") " pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" Jul 07 15:06:45.096254 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:45.096246 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dklt9\" (UniqueName: \"kubernetes.io/projected/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-kube-api-access-dklt9\") pod \"e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj\" (UID: \"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7\") " pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" Jul 07 15:06:45.096403 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:45.096314 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-istio-podinfo\") pod \"e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj\" (UID: \"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7\") " pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" Jul 07 15:06:45.096403 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:45.096321 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-credential-socket\") pod \"e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj\" (UID: \"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7\") " pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" Jul 07 15:06:45.096403 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:45.096371 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-istio-envoy\") pod \"e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj\" (UID: \"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7\") " pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" Jul 07 15:06:45.096551 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:45.096418 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-istiod-ca-cert\") pod \"e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj\" (UID: \"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7\") " pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" Jul 07 15:06:45.098581 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:45.098558 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-istio-envoy\") pod \"e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj\" (UID: \"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7\") " pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" Jul 07 15:06:45.098844 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:45.098825 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-istio-podinfo\") pod \"e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj\" (UID: \"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7\") " pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" Jul 07 15:06:45.104036 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:45.104017 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-istio-token\") pod \"e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj\" (UID: \"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7\") " pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" Jul 07 15:06:45.104036 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:45.104028 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dklt9\" (UniqueName: \"kubernetes.io/projected/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-kube-api-access-dklt9\") pod \"e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj\" (UID: \"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7\") " pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" Jul 07 15:06:45.234949 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:45.234856 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" Jul 07 15:06:45.369769 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:45.369731 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj"] Jul 07 15:06:45.371585 ip-10-0-136-19 kubenswrapper[2555]: W0707 15:06:45.371559 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0e494a2_d4b3_4922_b8b0_4578cca8a8b7.slice/crio-e8a09b23c8636c9c34aa2214a1499c39e967247bad5f5afcbef4fb3646662099 WatchSource:0}: Error finding container e8a09b23c8636c9c34aa2214a1499c39e967247bad5f5afcbef4fb3646662099: Status 404 returned error can't find the container with id e8a09b23c8636c9c34aa2214a1499c39e967247bad5f5afcbef4fb3646662099 Jul 07 15:06:45.373822 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:45.373786 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:06:45.373915 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:45.373856 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:06:45.373915 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:45.373886 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:06:46.262002 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:46.261967 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" event={"ID":"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7","Type":"ContainerStarted","Data":"6c7f58c92ac8cc30d8167e7eada6f1c39b792d8a4ac55b03b657a68ac0b64c94"} Jul 07 15:06:46.262442 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:46.262042 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" event={"ID":"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7","Type":"ContainerStarted","Data":"e8a09b23c8636c9c34aa2214a1499c39e967247bad5f5afcbef4fb3646662099"} Jul 07 15:06:46.285564 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:46.285515 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" podStartSLOduration=2.285500288 podStartE2EDuration="2.285500288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:06:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:06:46.283605042 +0000 UTC m=+1947.707791521" watchObservedRunningTime="2026-07-07 15:06:46.285500288 +0000 UTC m=+1947.709686764" Jul 07 15:06:47.235464 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:47.235433 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" Jul 07 15:06:47.240258 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:47.240236 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" Jul 07 15:06:47.269788 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:47.269739 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" Jul 07 15:06:47.270593 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:47.270579 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" Jul 07 15:06:48.411161 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:48.411131 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-06869d8e-67c5485fbf-l8ws6"] Jul 07 15:06:48.414824 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:48.414804 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-06869d8e-67c5485fbf-l8ws6" Jul 07 15:06:48.419047 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:48.419026 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-06869d8e-serving-cert\"" Jul 07 15:06:48.434460 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:48.434436 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-06869d8e-67c5485fbf-l8ws6"] Jul 07 15:06:48.529069 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:48.529027 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5t8l\" (UniqueName: \"kubernetes.io/projected/1ecd79cc-319c-46ee-b315-b9c5718e69fb-kube-api-access-p5t8l\") pod \"maas-api-e2e-mt-06869d8e-67c5485fbf-l8ws6\" (UID: \"1ecd79cc-319c-46ee-b315-b9c5718e69fb\") " pod="opendatahub/maas-api-e2e-mt-06869d8e-67c5485fbf-l8ws6" Jul 07 15:06:48.529262 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:48.529080 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ecd79cc-319c-46ee-b315-b9c5718e69fb-maas-api-tls\") pod \"maas-api-e2e-mt-06869d8e-67c5485fbf-l8ws6\" (UID: \"1ecd79cc-319c-46ee-b315-b9c5718e69fb\") " pod="opendatahub/maas-api-e2e-mt-06869d8e-67c5485fbf-l8ws6" Jul 07 15:06:48.529262 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:48.529210 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ecd79cc-319c-46ee-b315-b9c5718e69fb-openshift-service-ca\") pod \"maas-api-e2e-mt-06869d8e-67c5485fbf-l8ws6\" (UID: \"1ecd79cc-319c-46ee-b315-b9c5718e69fb\") " pod="opendatahub/maas-api-e2e-mt-06869d8e-67c5485fbf-l8ws6" Jul 07 15:06:48.630372 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:48.630336 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5t8l\" (UniqueName: \"kubernetes.io/projected/1ecd79cc-319c-46ee-b315-b9c5718e69fb-kube-api-access-p5t8l\") pod \"maas-api-e2e-mt-06869d8e-67c5485fbf-l8ws6\" (UID: \"1ecd79cc-319c-46ee-b315-b9c5718e69fb\") " pod="opendatahub/maas-api-e2e-mt-06869d8e-67c5485fbf-l8ws6" Jul 07 15:06:48.630574 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:48.630392 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ecd79cc-319c-46ee-b315-b9c5718e69fb-maas-api-tls\") pod \"maas-api-e2e-mt-06869d8e-67c5485fbf-l8ws6\" (UID: \"1ecd79cc-319c-46ee-b315-b9c5718e69fb\") " pod="opendatahub/maas-api-e2e-mt-06869d8e-67c5485fbf-l8ws6" Jul 07 15:06:48.630574 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:48.630505 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ecd79cc-319c-46ee-b315-b9c5718e69fb-openshift-service-ca\") pod \"maas-api-e2e-mt-06869d8e-67c5485fbf-l8ws6\" (UID: \"1ecd79cc-319c-46ee-b315-b9c5718e69fb\") " pod="opendatahub/maas-api-e2e-mt-06869d8e-67c5485fbf-l8ws6" Jul 07 15:06:48.630574 ip-10-0-136-19 kubenswrapper[2555]: E0707 15:06:48.630556 2555 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-06869d8e-serving-cert: secret "maas-api-e2e-mt-06869d8e-serving-cert" not found Jul 07 15:06:48.630717 ip-10-0-136-19 kubenswrapper[2555]: E0707 15:06:48.630613 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ecd79cc-319c-46ee-b315-b9c5718e69fb-maas-api-tls podName:1ecd79cc-319c-46ee-b315-b9c5718e69fb nodeName:}" failed. No retries permitted until 2026-07-07 15:06:49.130597642 +0000 UTC m=+1950.554784095 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1ecd79cc-319c-46ee-b315-b9c5718e69fb-maas-api-tls") pod "maas-api-e2e-mt-06869d8e-67c5485fbf-l8ws6" (UID: "1ecd79cc-319c-46ee-b315-b9c5718e69fb") : secret "maas-api-e2e-mt-06869d8e-serving-cert" not found Jul 07 15:06:48.631189 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:48.631170 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ecd79cc-319c-46ee-b315-b9c5718e69fb-openshift-service-ca\") pod \"maas-api-e2e-mt-06869d8e-67c5485fbf-l8ws6\" (UID: \"1ecd79cc-319c-46ee-b315-b9c5718e69fb\") " pod="opendatahub/maas-api-e2e-mt-06869d8e-67c5485fbf-l8ws6" Jul 07 15:06:48.645889 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:48.645866 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5t8l\" (UniqueName: \"kubernetes.io/projected/1ecd79cc-319c-46ee-b315-b9c5718e69fb-kube-api-access-p5t8l\") pod \"maas-api-e2e-mt-06869d8e-67c5485fbf-l8ws6\" (UID: \"1ecd79cc-319c-46ee-b315-b9c5718e69fb\") " pod="opendatahub/maas-api-e2e-mt-06869d8e-67c5485fbf-l8ws6" Jul 07 15:06:49.135056 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:49.135013 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ecd79cc-319c-46ee-b315-b9c5718e69fb-maas-api-tls\") pod \"maas-api-e2e-mt-06869d8e-67c5485fbf-l8ws6\" (UID: \"1ecd79cc-319c-46ee-b315-b9c5718e69fb\") " pod="opendatahub/maas-api-e2e-mt-06869d8e-67c5485fbf-l8ws6" Jul 07 15:06:49.137461 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:49.137432 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ecd79cc-319c-46ee-b315-b9c5718e69fb-maas-api-tls\") pod \"maas-api-e2e-mt-06869d8e-67c5485fbf-l8ws6\" (UID: \"1ecd79cc-319c-46ee-b315-b9c5718e69fb\") " pod="opendatahub/maas-api-e2e-mt-06869d8e-67c5485fbf-l8ws6" Jul 07 15:06:49.325645 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:49.325614 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-06869d8e-67c5485fbf-l8ws6" Jul 07 15:06:49.481529 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:49.481504 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-06869d8e-67c5485fbf-l8ws6"] Jul 07 15:06:49.483477 ip-10-0-136-19 kubenswrapper[2555]: W0707 15:06:49.483452 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ecd79cc_319c_46ee_b315_b9c5718e69fb.slice/crio-9a334b2e43c4972ccce28dff65e8dfd5579e51633853a021a26026c288f22a20 WatchSource:0}: Error finding container 9a334b2e43c4972ccce28dff65e8dfd5579e51633853a021a26026c288f22a20: Status 404 returned error can't find the container with id 9a334b2e43c4972ccce28dff65e8dfd5579e51633853a021a26026c288f22a20 Jul 07 15:06:50.286244 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:50.286155 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-06869d8e-67c5485fbf-l8ws6" event={"ID":"1ecd79cc-319c-46ee-b315-b9c5718e69fb","Type":"ContainerStarted","Data":"eab17ffd2f3c402ba86f9a6f8e9ac26131f541b2817fa793a0d0e66011305385"} Jul 07 15:06:50.286244 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:50.286212 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-06869d8e-67c5485fbf-l8ws6" event={"ID":"1ecd79cc-319c-46ee-b315-b9c5718e69fb","Type":"ContainerStarted","Data":"9a334b2e43c4972ccce28dff65e8dfd5579e51633853a021a26026c288f22a20"} Jul 07 15:06:50.286461 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:50.286263 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-06869d8e-67c5485fbf-l8ws6" Jul 07 15:06:50.311493 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:50.311329 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-06869d8e-67c5485fbf-l8ws6" podStartSLOduration=1.822973219 podStartE2EDuration="2.311309696s" totalImagesPullingTime="488.336477ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:06:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:06:50.31100037 +0000 UTC m=+1951.735186845" watchObservedRunningTime="2026-07-07 15:06:50.311309696 +0000 UTC m=+1951.735496173" Jul 07 15:06:56.296348 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:06:56.296322 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-06869d8e-67c5485fbf-l8ws6" Jul 07 15:07:00.537688 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:00.537598 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-06869d8e-67c5485fbf-l8ws6"] Jul 07 15:07:00.538164 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:00.537857 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-06869d8e-67c5485fbf-l8ws6" podUID="1ecd79cc-319c-46ee-b315-b9c5718e69fb" containerName="maas-api" containerID="cri-o://eab17ffd2f3c402ba86f9a6f8e9ac26131f541b2817fa793a0d0e66011305385" gracePeriod=30 Jul 07 15:07:00.795658 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:00.795596 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-06869d8e-67c5485fbf-l8ws6" Jul 07 15:07:00.842972 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:00.842936 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ecd79cc-319c-46ee-b315-b9c5718e69fb-openshift-service-ca\") pod \"1ecd79cc-319c-46ee-b315-b9c5718e69fb\" (UID: \"1ecd79cc-319c-46ee-b315-b9c5718e69fb\") " Jul 07 15:07:00.843170 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:00.843015 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5t8l\" (UniqueName: \"kubernetes.io/projected/1ecd79cc-319c-46ee-b315-b9c5718e69fb-kube-api-access-p5t8l\") pod \"1ecd79cc-319c-46ee-b315-b9c5718e69fb\" (UID: \"1ecd79cc-319c-46ee-b315-b9c5718e69fb\") " Jul 07 15:07:00.843170 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:00.843077 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ecd79cc-319c-46ee-b315-b9c5718e69fb-maas-api-tls\") pod \"1ecd79cc-319c-46ee-b315-b9c5718e69fb\" (UID: \"1ecd79cc-319c-46ee-b315-b9c5718e69fb\") " Jul 07 15:07:00.843378 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:00.843353 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ecd79cc-319c-46ee-b315-b9c5718e69fb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1ecd79cc-319c-46ee-b315-b9c5718e69fb" (UID: "1ecd79cc-319c-46ee-b315-b9c5718e69fb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:07:00.845147 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:00.845114 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ecd79cc-319c-46ee-b315-b9c5718e69fb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1ecd79cc-319c-46ee-b315-b9c5718e69fb" (UID: "1ecd79cc-319c-46ee-b315-b9c5718e69fb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:07:00.845266 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:00.845183 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ecd79cc-319c-46ee-b315-b9c5718e69fb-kube-api-access-p5t8l" (OuterVolumeSpecName: "kube-api-access-p5t8l") pod "1ecd79cc-319c-46ee-b315-b9c5718e69fb" (UID: "1ecd79cc-319c-46ee-b315-b9c5718e69fb"). InnerVolumeSpecName "kube-api-access-p5t8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:07:00.944487 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:00.944448 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5t8l\" (UniqueName: \"kubernetes.io/projected/1ecd79cc-319c-46ee-b315-b9c5718e69fb-kube-api-access-p5t8l\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:07:00.944487 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:00.944478 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ecd79cc-319c-46ee-b315-b9c5718e69fb-maas-api-tls\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:07:00.944487 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:00.944488 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ecd79cc-319c-46ee-b315-b9c5718e69fb-openshift-service-ca\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:07:01.338334 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:01.338300 2555 generic.go:356] "Generic (PLEG): container finished" podID="1ecd79cc-319c-46ee-b315-b9c5718e69fb" containerID="eab17ffd2f3c402ba86f9a6f8e9ac26131f541b2817fa793a0d0e66011305385" exitCode=0 Jul 07 15:07:01.338541 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:01.338335 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-06869d8e-67c5485fbf-l8ws6" Jul 07 15:07:01.338541 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:01.338381 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-06869d8e-67c5485fbf-l8ws6" event={"ID":"1ecd79cc-319c-46ee-b315-b9c5718e69fb","Type":"ContainerDied","Data":"eab17ffd2f3c402ba86f9a6f8e9ac26131f541b2817fa793a0d0e66011305385"} Jul 07 15:07:01.338541 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:01.338438 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-06869d8e-67c5485fbf-l8ws6" event={"ID":"1ecd79cc-319c-46ee-b315-b9c5718e69fb","Type":"ContainerDied","Data":"9a334b2e43c4972ccce28dff65e8dfd5579e51633853a021a26026c288f22a20"} Jul 07 15:07:01.338541 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:01.338468 2555 scope.go:117] "RemoveContainer" containerID="eab17ffd2f3c402ba86f9a6f8e9ac26131f541b2817fa793a0d0e66011305385" Jul 07 15:07:01.347736 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:01.347712 2555 scope.go:117] "RemoveContainer" containerID="eab17ffd2f3c402ba86f9a6f8e9ac26131f541b2817fa793a0d0e66011305385" Jul 07 15:07:01.348064 ip-10-0-136-19 kubenswrapper[2555]: E0707 15:07:01.348036 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eab17ffd2f3c402ba86f9a6f8e9ac26131f541b2817fa793a0d0e66011305385\": container with ID starting with eab17ffd2f3c402ba86f9a6f8e9ac26131f541b2817fa793a0d0e66011305385 not found: ID does not exist" containerID="eab17ffd2f3c402ba86f9a6f8e9ac26131f541b2817fa793a0d0e66011305385" Jul 07 15:07:01.348161 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:01.348073 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eab17ffd2f3c402ba86f9a6f8e9ac26131f541b2817fa793a0d0e66011305385"} err="failed to get container status \"eab17ffd2f3c402ba86f9a6f8e9ac26131f541b2817fa793a0d0e66011305385\": rpc error: code = NotFound desc = could not find container \"eab17ffd2f3c402ba86f9a6f8e9ac26131f541b2817fa793a0d0e66011305385\": container with ID starting with eab17ffd2f3c402ba86f9a6f8e9ac26131f541b2817fa793a0d0e66011305385 not found: ID does not exist" Jul 07 15:07:01.357540 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:01.357514 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-06869d8e-67c5485fbf-l8ws6"] Jul 07 15:07:01.359204 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:01.359181 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-06869d8e-67c5485fbf-l8ws6"] Jul 07 15:07:02.818097 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:02.817018 2555 patch_prober.go:28] interesting pod/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": read tcp 10.133.0.2:55982->10.133.0.40:15021: read: connection reset by peer" start-of-body= Jul 07 15:07:02.818097 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:02.817082 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" podUID="e0e494a2-d4b3-4922-b8b0-4578cca8a8b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": read tcp 10.133.0.2:55982->10.133.0.40:15021: read: connection reset by peer" Jul 07 15:07:03.042263 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:03.042192 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ecd79cc-319c-46ee-b315-b9c5718e69fb" path="/var/lib/kubelet/pods/1ecd79cc-319c-46ee-b315-b9c5718e69fb/volumes" Jul 07 15:07:03.350131 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:03.350099 2555 generic.go:356] "Generic (PLEG): container finished" podID="e0e494a2-d4b3-4922-b8b0-4578cca8a8b7" containerID="6c7f58c92ac8cc30d8167e7eada6f1c39b792d8a4ac55b03b657a68ac0b64c94" exitCode=0 Jul 07 15:07:03.350337 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:03.350173 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" event={"ID":"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7","Type":"ContainerDied","Data":"6c7f58c92ac8cc30d8167e7eada6f1c39b792d8a4ac55b03b657a68ac0b64c94"} Jul 07 15:07:03.350683 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:03.350660 2555 scope.go:117] "RemoveContainer" containerID="6c7f58c92ac8cc30d8167e7eada6f1c39b792d8a4ac55b03b657a68ac0b64c94" Jul 07 15:07:03.351778 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:03.351737 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:07:03.351851 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:03.351822 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:07:03.351892 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:03.351859 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:07:04.356499 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:04.356464 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" event={"ID":"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7","Type":"ContainerStarted","Data":"90dc04b1dd09c04efe1ffbc2284271f6909bbc13536a670c3fad960f31fc1b43"} Jul 07 15:07:05.235622 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:05.235587 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" Jul 07 15:07:05.238996 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:05.238971 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" Jul 07 15:07:05.360998 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:05.360963 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" Jul 07 15:07:05.362072 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:05.362055 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" Jul 07 15:07:07.077413 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:07.077379 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj"] Jul 07 15:07:07.370231 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:07.370145 2555 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" secret="" err="secret \"e2e-mt-06869d8e-openshift-default-dockercfg-65kjb\" not found" Jul 07 15:07:08.374979 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:08.374921 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" podUID="e0e494a2-d4b3-4922-b8b0-4578cca8a8b7" containerName="istio-proxy" containerID="cri-o://90dc04b1dd09c04efe1ffbc2284271f6909bbc13536a670c3fad960f31fc1b43" gracePeriod=30 Jul 07 15:07:13.620627 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:13.620604 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" Jul 07 15:07:13.767730 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:13.767642 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-istio-podinfo\") pod \"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7\" (UID: \"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7\") " Jul 07 15:07:13.767730 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:13.767708 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-istio-envoy\") pod \"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7\" (UID: \"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7\") " Jul 07 15:07:13.768002 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:13.767738 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-credential-socket\") pod \"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7\" (UID: \"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7\") " Jul 07 15:07:13.768002 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:13.767794 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-workload-certs\") pod \"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7\" (UID: \"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7\") " Jul 07 15:07:13.768002 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:13.767848 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-istiod-ca-cert\") pod \"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7\" (UID: \"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7\") " Jul 07 15:07:13.768002 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:13.767899 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dklt9\" (UniqueName: \"kubernetes.io/projected/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-kube-api-access-dklt9\") pod \"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7\" (UID: \"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7\") " Jul 07 15:07:13.768002 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:13.767938 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-istio-data\") pod \"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7\" (UID: \"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7\") " Jul 07 15:07:13.768002 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:13.767968 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-workload-socket\") pod \"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7\" (UID: \"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7\") " Jul 07 15:07:13.768305 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:13.768028 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-istio-token\") pod \"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7\" (UID: \"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7\") " Jul 07 15:07:13.768305 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:13.768151 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e0e494a2-d4b3-4922-b8b0-4578cca8a8b7" (UID: "e0e494a2-d4b3-4922-b8b0-4578cca8a8b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:07:13.768305 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:13.768163 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e0e494a2-d4b3-4922-b8b0-4578cca8a8b7" (UID: "e0e494a2-d4b3-4922-b8b0-4578cca8a8b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:07:13.768498 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:13.768463 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e0e494a2-d4b3-4922-b8b0-4578cca8a8b7" (UID: "e0e494a2-d4b3-4922-b8b0-4578cca8a8b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:07:13.768803 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:13.768689 2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-credential-socket\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:07:13.768803 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:13.768718 2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-workload-certs\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:07:13.768803 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:13.768733 2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-istiod-ca-cert\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:07:13.768803 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:13.768729 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e0e494a2-d4b3-4922-b8b0-4578cca8a8b7" (UID: "e0e494a2-d4b3-4922-b8b0-4578cca8a8b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:07:13.769484 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:13.769451 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-istio-data" (OuterVolumeSpecName: "istio-data") pod "e0e494a2-d4b3-4922-b8b0-4578cca8a8b7" (UID: "e0e494a2-d4b3-4922-b8b0-4578cca8a8b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:07:13.770255 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:13.770224 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e0e494a2-d4b3-4922-b8b0-4578cca8a8b7" (UID: "e0e494a2-d4b3-4922-b8b0-4578cca8a8b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 15:07:13.770668 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:13.770636 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-kube-api-access-dklt9" (OuterVolumeSpecName: "kube-api-access-dklt9") pod "e0e494a2-d4b3-4922-b8b0-4578cca8a8b7" (UID: "e0e494a2-d4b3-4922-b8b0-4578cca8a8b7"). InnerVolumeSpecName "kube-api-access-dklt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:07:13.770668 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:13.770659 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e0e494a2-d4b3-4922-b8b0-4578cca8a8b7" (UID: "e0e494a2-d4b3-4922-b8b0-4578cca8a8b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:07:13.770837 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:13.770675 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-istio-token" (OuterVolumeSpecName: "istio-token") pod "e0e494a2-d4b3-4922-b8b0-4578cca8a8b7" (UID: "e0e494a2-d4b3-4922-b8b0-4578cca8a8b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:07:13.869317 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:13.869285 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dklt9\" (UniqueName: \"kubernetes.io/projected/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-kube-api-access-dklt9\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:07:13.869317 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:13.869313 2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-istio-data\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:07:13.869317 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:13.869324 2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-workload-socket\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:07:13.869562 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:13.869332 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-istio-token\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:07:13.869562 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:13.869341 2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-istio-podinfo\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:07:13.869562 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:13.869348 2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7-istio-envoy\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:07:14.405443 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:14.405407 2555 generic.go:356] "Generic (PLEG): container finished" podID="e0e494a2-d4b3-4922-b8b0-4578cca8a8b7" containerID="90dc04b1dd09c04efe1ffbc2284271f6909bbc13536a670c3fad960f31fc1b43" exitCode=0 Jul 07 15:07:14.405655 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:14.405465 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" Jul 07 15:07:14.405655 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:14.405460 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" event={"ID":"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7","Type":"ContainerDied","Data":"90dc04b1dd09c04efe1ffbc2284271f6909bbc13536a670c3fad960f31fc1b43"} Jul 07 15:07:14.405655 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:14.405535 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj" event={"ID":"e0e494a2-d4b3-4922-b8b0-4578cca8a8b7","Type":"ContainerDied","Data":"e8a09b23c8636c9c34aa2214a1499c39e967247bad5f5afcbef4fb3646662099"} Jul 07 15:07:14.405655 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:14.405566 2555 scope.go:117] "RemoveContainer" containerID="90dc04b1dd09c04efe1ffbc2284271f6909bbc13536a670c3fad960f31fc1b43" Jul 07 15:07:14.415102 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:14.415086 2555 scope.go:117] "RemoveContainer" containerID="6c7f58c92ac8cc30d8167e7eada6f1c39b792d8a4ac55b03b657a68ac0b64c94" Jul 07 15:07:14.423037 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:14.423023 2555 scope.go:117] "RemoveContainer" containerID="90dc04b1dd09c04efe1ffbc2284271f6909bbc13536a670c3fad960f31fc1b43" Jul 07 15:07:14.423273 ip-10-0-136-19 kubenswrapper[2555]: E0707 15:07:14.423256 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90dc04b1dd09c04efe1ffbc2284271f6909bbc13536a670c3fad960f31fc1b43\": container with ID starting with 90dc04b1dd09c04efe1ffbc2284271f6909bbc13536a670c3fad960f31fc1b43 not found: ID does not exist" containerID="90dc04b1dd09c04efe1ffbc2284271f6909bbc13536a670c3fad960f31fc1b43" Jul 07 15:07:14.423316 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:14.423281 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90dc04b1dd09c04efe1ffbc2284271f6909bbc13536a670c3fad960f31fc1b43"} err="failed to get container status \"90dc04b1dd09c04efe1ffbc2284271f6909bbc13536a670c3fad960f31fc1b43\": rpc error: code = NotFound desc = could not find container \"90dc04b1dd09c04efe1ffbc2284271f6909bbc13536a670c3fad960f31fc1b43\": container with ID starting with 90dc04b1dd09c04efe1ffbc2284271f6909bbc13536a670c3fad960f31fc1b43 not found: ID does not exist" Jul 07 15:07:14.423316 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:14.423298 2555 scope.go:117] "RemoveContainer" containerID="6c7f58c92ac8cc30d8167e7eada6f1c39b792d8a4ac55b03b657a68ac0b64c94" Jul 07 15:07:14.423517 ip-10-0-136-19 kubenswrapper[2555]: E0707 15:07:14.423500 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c7f58c92ac8cc30d8167e7eada6f1c39b792d8a4ac55b03b657a68ac0b64c94\": container with ID starting with 6c7f58c92ac8cc30d8167e7eada6f1c39b792d8a4ac55b03b657a68ac0b64c94 not found: ID does not exist" containerID="6c7f58c92ac8cc30d8167e7eada6f1c39b792d8a4ac55b03b657a68ac0b64c94" Jul 07 15:07:14.423577 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:14.423525 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c7f58c92ac8cc30d8167e7eada6f1c39b792d8a4ac55b03b657a68ac0b64c94"} err="failed to get container status \"6c7f58c92ac8cc30d8167e7eada6f1c39b792d8a4ac55b03b657a68ac0b64c94\": rpc error: code = NotFound desc = could not find container \"6c7f58c92ac8cc30d8167e7eada6f1c39b792d8a4ac55b03b657a68ac0b64c94\": container with ID starting with 6c7f58c92ac8cc30d8167e7eada6f1c39b792d8a4ac55b03b657a68ac0b64c94 not found: ID does not exist" Jul 07 15:07:14.428650 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:14.428630 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj"] Jul 07 15:07:14.433207 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:14.433189 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-06869d8e-openshift-default-6dc44575f5-h6vwj"] Jul 07 15:07:15.042784 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:15.042736 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0e494a2-d4b3-4922-b8b0-4578cca8a8b7" path="/var/lib/kubelet/pods/e0e494a2-d4b3-4922-b8b0-4578cca8a8b7/volumes" Jul 07 15:07:26.824127 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:26.824087 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx"] Jul 07 15:07:26.824550 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:26.824476 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ecd79cc-319c-46ee-b315-b9c5718e69fb" containerName="maas-api" Jul 07 15:07:26.824550 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:26.824488 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ecd79cc-319c-46ee-b315-b9c5718e69fb" containerName="maas-api" Jul 07 15:07:26.824550 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:26.824499 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0e494a2-d4b3-4922-b8b0-4578cca8a8b7" containerName="istio-proxy" Jul 07 15:07:26.824550 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:26.824505 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0e494a2-d4b3-4922-b8b0-4578cca8a8b7" containerName="istio-proxy" Jul 07 15:07:26.824550 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:26.824512 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0e494a2-d4b3-4922-b8b0-4578cca8a8b7" containerName="istio-proxy" Jul 07 15:07:26.824550 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:26.824518 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0e494a2-d4b3-4922-b8b0-4578cca8a8b7" containerName="istio-proxy" Jul 07 15:07:26.824772 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:26.824580 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ecd79cc-319c-46ee-b315-b9c5718e69fb" containerName="maas-api" Jul 07 15:07:26.824772 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:26.824589 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0e494a2-d4b3-4922-b8b0-4578cca8a8b7" containerName="istio-proxy" Jul 07 15:07:26.824772 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:26.824595 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0e494a2-d4b3-4922-b8b0-4578cca8a8b7" containerName="istio-proxy" Jul 07 15:07:26.827514 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:26.827491 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" Jul 07 15:07:26.830286 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:26.830257 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-09af33f3-openshift-default-dockercfg-rx2fp\"" Jul 07 15:07:26.843654 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:26.843622 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx"] Jul 07 15:07:26.883482 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:26.883432 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pggn2\" (UniqueName: \"kubernetes.io/projected/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-kube-api-access-pggn2\") pod \"e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx\" (UID: \"c3aebc7d-b6f0-4e4a-bd51-1373175105a8\") " pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" Jul 07 15:07:26.883482 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:26.883481 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-istio-token\") pod \"e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx\" (UID: \"c3aebc7d-b6f0-4e4a-bd51-1373175105a8\") " pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" Jul 07 15:07:26.883711 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:26.883507 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-istio-podinfo\") pod \"e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx\" (UID: \"c3aebc7d-b6f0-4e4a-bd51-1373175105a8\") " pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" Jul 07 15:07:26.883711 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:26.883633 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-istiod-ca-cert\") pod \"e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx\" (UID: \"c3aebc7d-b6f0-4e4a-bd51-1373175105a8\") " pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" Jul 07 15:07:26.883711 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:26.883678 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-istio-envoy\") pod \"e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx\" (UID: \"c3aebc7d-b6f0-4e4a-bd51-1373175105a8\") " pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" Jul 07 15:07:26.883845 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:26.883727 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-workload-socket\") pod \"e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx\" (UID: \"c3aebc7d-b6f0-4e4a-bd51-1373175105a8\") " pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" Jul 07 15:07:26.883845 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:26.883745 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-credential-socket\") pod \"e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx\" (UID: \"c3aebc7d-b6f0-4e4a-bd51-1373175105a8\") " pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" Jul 07 15:07:26.883845 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:26.883789 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-workload-certs\") pod \"e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx\" (UID: \"c3aebc7d-b6f0-4e4a-bd51-1373175105a8\") " pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" Jul 07 15:07:26.883845 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:26.883811 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-istio-data\") pod \"e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx\" (UID: \"c3aebc7d-b6f0-4e4a-bd51-1373175105a8\") " pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" Jul 07 15:07:26.984673 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:26.984636 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-istiod-ca-cert\") pod \"e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx\" (UID: \"c3aebc7d-b6f0-4e4a-bd51-1373175105a8\") " pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" Jul 07 15:07:26.984920 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:26.984693 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-istio-envoy\") pod \"e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx\" (UID: \"c3aebc7d-b6f0-4e4a-bd51-1373175105a8\") " pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" Jul 07 15:07:26.984920 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:26.984741 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-workload-socket\") pod \"e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx\" (UID: \"c3aebc7d-b6f0-4e4a-bd51-1373175105a8\") " pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" Jul 07 15:07:26.984920 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:26.984794 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-credential-socket\") pod \"e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx\" (UID: \"c3aebc7d-b6f0-4e4a-bd51-1373175105a8\") " pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" Jul 07 15:07:26.984920 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:26.984822 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-workload-certs\") pod \"e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx\" (UID: \"c3aebc7d-b6f0-4e4a-bd51-1373175105a8\") " pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" Jul 07 15:07:26.984920 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:26.984842 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-istio-data\") pod \"e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx\" (UID: \"c3aebc7d-b6f0-4e4a-bd51-1373175105a8\") " pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" Jul 07 15:07:26.984920 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:26.984877 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pggn2\" (UniqueName: \"kubernetes.io/projected/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-kube-api-access-pggn2\") pod \"e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx\" (UID: \"c3aebc7d-b6f0-4e4a-bd51-1373175105a8\") " pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" Jul 07 15:07:26.984920 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:26.984904 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-istio-token\") pod \"e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx\" (UID: \"c3aebc7d-b6f0-4e4a-bd51-1373175105a8\") " pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" Jul 07 15:07:26.985286 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:26.985147 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-istio-podinfo\") pod \"e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx\" (UID: \"c3aebc7d-b6f0-4e4a-bd51-1373175105a8\") " pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" Jul 07 15:07:26.985286 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:26.985254 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-workload-socket\") pod \"e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx\" (UID: \"c3aebc7d-b6f0-4e4a-bd51-1373175105a8\") " pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" Jul 07 15:07:26.985396 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:26.985291 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-credential-socket\") pod \"e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx\" (UID: \"c3aebc7d-b6f0-4e4a-bd51-1373175105a8\") " pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" Jul 07 15:07:26.985396 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:26.985342 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-istio-data\") pod \"e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx\" (UID: \"c3aebc7d-b6f0-4e4a-bd51-1373175105a8\") " pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" Jul 07 15:07:26.985396 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:26.985359 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-workload-certs\") pod \"e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx\" (UID: \"c3aebc7d-b6f0-4e4a-bd51-1373175105a8\") " pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" Jul 07 15:07:26.985564 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:26.985465 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-istiod-ca-cert\") pod \"e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx\" (UID: \"c3aebc7d-b6f0-4e4a-bd51-1373175105a8\") " pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" Jul 07 15:07:26.987503 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:26.987480 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-istio-envoy\") pod \"e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx\" (UID: \"c3aebc7d-b6f0-4e4a-bd51-1373175105a8\") " pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" Jul 07 15:07:26.987730 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:26.987712 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-istio-podinfo\") pod \"e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx\" (UID: \"c3aebc7d-b6f0-4e4a-bd51-1373175105a8\") " pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" Jul 07 15:07:26.996429 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:26.996408 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pggn2\" (UniqueName: \"kubernetes.io/projected/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-kube-api-access-pggn2\") pod \"e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx\" (UID: \"c3aebc7d-b6f0-4e4a-bd51-1373175105a8\") " pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" Jul 07 15:07:26.996559 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:26.996457 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-istio-token\") pod \"e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx\" (UID: \"c3aebc7d-b6f0-4e4a-bd51-1373175105a8\") " pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" Jul 07 15:07:27.138510 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:27.138479 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" Jul 07 15:07:27.276052 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:27.276028 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx"] Jul 07 15:07:27.278275 ip-10-0-136-19 kubenswrapper[2555]: W0707 15:07:27.278253 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3aebc7d_b6f0_4e4a_bd51_1373175105a8.slice/crio-02946070b3e49107b59c43da7870db0266891835ddfc8b71db767b0fea2783f1 WatchSource:0}: Error finding container 02946070b3e49107b59c43da7870db0266891835ddfc8b71db767b0fea2783f1: Status 404 returned error can't find the container with id 02946070b3e49107b59c43da7870db0266891835ddfc8b71db767b0fea2783f1 Jul 07 15:07:27.280295 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:27.280266 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:07:27.280390 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:27.280328 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:07:27.280390 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:27.280357 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:07:27.466335 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:27.466240 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" event={"ID":"c3aebc7d-b6f0-4e4a-bd51-1373175105a8","Type":"ContainerStarted","Data":"71edaefa0a005117e1444a2c7c609ec6541fe2e8c239ab1931312a40fd66a787"} Jul 07 15:07:27.466335 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:27.466300 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" event={"ID":"c3aebc7d-b6f0-4e4a-bd51-1373175105a8","Type":"ContainerStarted","Data":"02946070b3e49107b59c43da7870db0266891835ddfc8b71db767b0fea2783f1"} Jul 07 15:07:27.491414 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:27.491361 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" podStartSLOduration=1.49134514 podStartE2EDuration="1.49134514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:07:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:07:27.489403507 +0000 UTC m=+1988.913589985" watchObservedRunningTime="2026-07-07 15:07:27.49134514 +0000 UTC m=+1988.915531615" Jul 07 15:07:28.139382 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:28.139346 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" Jul 07 15:07:28.144306 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:28.144281 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" Jul 07 15:07:28.471288 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:28.471194 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" Jul 07 15:07:28.472328 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:28.472309 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" Jul 07 15:07:29.612495 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:29.612458 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-09af33f3-5979f57f56-k2gg2"] Jul 07 15:07:29.615647 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:29.615623 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-09af33f3-5979f57f56-k2gg2" Jul 07 15:07:29.618519 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:29.618490 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-09af33f3-serving-cert\"" Jul 07 15:07:29.635080 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:29.635040 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-09af33f3-5979f57f56-k2gg2"] Jul 07 15:07:29.712519 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:29.712470 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/734daae0-3ff2-48fd-b2d2-d8eb05d08e63-openshift-service-ca\") pod \"maas-api-e2e-mt-09af33f3-5979f57f56-k2gg2\" (UID: \"734daae0-3ff2-48fd-b2d2-d8eb05d08e63\") " pod="opendatahub/maas-api-e2e-mt-09af33f3-5979f57f56-k2gg2" Jul 07 15:07:29.712708 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:29.712617 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npsfm\" (UniqueName: \"kubernetes.io/projected/734daae0-3ff2-48fd-b2d2-d8eb05d08e63-kube-api-access-npsfm\") pod \"maas-api-e2e-mt-09af33f3-5979f57f56-k2gg2\" (UID: \"734daae0-3ff2-48fd-b2d2-d8eb05d08e63\") " pod="opendatahub/maas-api-e2e-mt-09af33f3-5979f57f56-k2gg2" Jul 07 15:07:29.712831 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:29.712715 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/734daae0-3ff2-48fd-b2d2-d8eb05d08e63-maas-api-tls\") pod \"maas-api-e2e-mt-09af33f3-5979f57f56-k2gg2\" (UID: \"734daae0-3ff2-48fd-b2d2-d8eb05d08e63\") " pod="opendatahub/maas-api-e2e-mt-09af33f3-5979f57f56-k2gg2" Jul 07 15:07:29.813384 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:29.813351 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npsfm\" (UniqueName: \"kubernetes.io/projected/734daae0-3ff2-48fd-b2d2-d8eb05d08e63-kube-api-access-npsfm\") pod \"maas-api-e2e-mt-09af33f3-5979f57f56-k2gg2\" (UID: \"734daae0-3ff2-48fd-b2d2-d8eb05d08e63\") " pod="opendatahub/maas-api-e2e-mt-09af33f3-5979f57f56-k2gg2" Jul 07 15:07:29.813544 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:29.813408 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/734daae0-3ff2-48fd-b2d2-d8eb05d08e63-maas-api-tls\") pod \"maas-api-e2e-mt-09af33f3-5979f57f56-k2gg2\" (UID: \"734daae0-3ff2-48fd-b2d2-d8eb05d08e63\") " pod="opendatahub/maas-api-e2e-mt-09af33f3-5979f57f56-k2gg2" Jul 07 15:07:29.813544 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:29.813442 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/734daae0-3ff2-48fd-b2d2-d8eb05d08e63-openshift-service-ca\") pod \"maas-api-e2e-mt-09af33f3-5979f57f56-k2gg2\" (UID: \"734daae0-3ff2-48fd-b2d2-d8eb05d08e63\") " pod="opendatahub/maas-api-e2e-mt-09af33f3-5979f57f56-k2gg2" Jul 07 15:07:29.813655 ip-10-0-136-19 kubenswrapper[2555]: E0707 15:07:29.813564 2555 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-09af33f3-serving-cert: secret "maas-api-e2e-mt-09af33f3-serving-cert" not found Jul 07 15:07:29.813655 ip-10-0-136-19 kubenswrapper[2555]: E0707 15:07:29.813632 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/734daae0-3ff2-48fd-b2d2-d8eb05d08e63-maas-api-tls podName:734daae0-3ff2-48fd-b2d2-d8eb05d08e63 nodeName:}" failed. No retries permitted until 2026-07-07 15:07:30.313611884 +0000 UTC m=+1991.737798346 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/734daae0-3ff2-48fd-b2d2-d8eb05d08e63-maas-api-tls") pod "maas-api-e2e-mt-09af33f3-5979f57f56-k2gg2" (UID: "734daae0-3ff2-48fd-b2d2-d8eb05d08e63") : secret "maas-api-e2e-mt-09af33f3-serving-cert" not found Jul 07 15:07:29.814154 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:29.814136 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/734daae0-3ff2-48fd-b2d2-d8eb05d08e63-openshift-service-ca\") pod \"maas-api-e2e-mt-09af33f3-5979f57f56-k2gg2\" (UID: \"734daae0-3ff2-48fd-b2d2-d8eb05d08e63\") " pod="opendatahub/maas-api-e2e-mt-09af33f3-5979f57f56-k2gg2" Jul 07 15:07:29.825658 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:29.825630 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npsfm\" (UniqueName: \"kubernetes.io/projected/734daae0-3ff2-48fd-b2d2-d8eb05d08e63-kube-api-access-npsfm\") pod \"maas-api-e2e-mt-09af33f3-5979f57f56-k2gg2\" (UID: \"734daae0-3ff2-48fd-b2d2-d8eb05d08e63\") " pod="opendatahub/maas-api-e2e-mt-09af33f3-5979f57f56-k2gg2" Jul 07 15:07:30.319080 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:30.319036 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/734daae0-3ff2-48fd-b2d2-d8eb05d08e63-maas-api-tls\") pod \"maas-api-e2e-mt-09af33f3-5979f57f56-k2gg2\" (UID: \"734daae0-3ff2-48fd-b2d2-d8eb05d08e63\") " pod="opendatahub/maas-api-e2e-mt-09af33f3-5979f57f56-k2gg2" Jul 07 15:07:30.321535 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:30.321508 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/734daae0-3ff2-48fd-b2d2-d8eb05d08e63-maas-api-tls\") pod \"maas-api-e2e-mt-09af33f3-5979f57f56-k2gg2\" (UID: \"734daae0-3ff2-48fd-b2d2-d8eb05d08e63\") " pod="opendatahub/maas-api-e2e-mt-09af33f3-5979f57f56-k2gg2" Jul 07 15:07:30.527103 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:30.527070 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-09af33f3-5979f57f56-k2gg2" Jul 07 15:07:30.728293 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:30.728268 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-09af33f3-5979f57f56-k2gg2"] Jul 07 15:07:30.730616 ip-10-0-136-19 kubenswrapper[2555]: W0707 15:07:30.730585 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod734daae0_3ff2_48fd_b2d2_d8eb05d08e63.slice/crio-fd2b804bf4985d4555ba97bcf2d9a0a26358baeb11feeed456def6ad140d9952 WatchSource:0}: Error finding container fd2b804bf4985d4555ba97bcf2d9a0a26358baeb11feeed456def6ad140d9952: Status 404 returned error can't find the container with id fd2b804bf4985d4555ba97bcf2d9a0a26358baeb11feeed456def6ad140d9952 Jul 07 15:07:31.491512 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:31.491420 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-09af33f3-5979f57f56-k2gg2" event={"ID":"734daae0-3ff2-48fd-b2d2-d8eb05d08e63","Type":"ContainerStarted","Data":"de9e65b89d2bb50d8c40a4b9db2e1a4774129e742cc1db12ed4e074541f35367"} Jul 07 15:07:31.491512 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:31.491473 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-09af33f3-5979f57f56-k2gg2" event={"ID":"734daae0-3ff2-48fd-b2d2-d8eb05d08e63","Type":"ContainerStarted","Data":"fd2b804bf4985d4555ba97bcf2d9a0a26358baeb11feeed456def6ad140d9952"} Jul 07 15:07:31.491512 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:31.491501 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-09af33f3-5979f57f56-k2gg2" Jul 07 15:07:37.500850 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:37.500821 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-09af33f3-5979f57f56-k2gg2" Jul 07 15:07:37.519060 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:37.518992 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-09af33f3-5979f57f56-k2gg2" podStartSLOduration=8.050964231 podStartE2EDuration="8.518973347s" totalImagesPullingTime="468.009116ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:07:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:07:31.513539974 +0000 UTC m=+1992.937726450" watchObservedRunningTime="2026-07-07 15:07:37.518973347 +0000 UTC m=+1998.943159826" Jul 07 15:07:43.473133 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:43.473038 2555 patch_prober.go:28] interesting pod/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 07 15:07:43.473133 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:43.473092 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" podUID="c3aebc7d-b6f0-4e4a-bd51-1373175105a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 07 15:07:43.547900 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:43.547859 2555 generic.go:356] "Generic (PLEG): container finished" podID="c3aebc7d-b6f0-4e4a-bd51-1373175105a8" containerID="71edaefa0a005117e1444a2c7c609ec6541fe2e8c239ab1931312a40fd66a787" exitCode=0 Jul 07 15:07:43.548062 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:43.547916 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" event={"ID":"c3aebc7d-b6f0-4e4a-bd51-1373175105a8","Type":"ContainerDied","Data":"71edaefa0a005117e1444a2c7c609ec6541fe2e8c239ab1931312a40fd66a787"} Jul 07 15:07:43.548381 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:43.548365 2555 scope.go:117] "RemoveContainer" containerID="71edaefa0a005117e1444a2c7c609ec6541fe2e8c239ab1931312a40fd66a787" Jul 07 15:07:43.549387 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:43.549359 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:07:43.549481 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:43.549441 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:07:43.549481 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:43.549473 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:07:44.561581 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:44.561541 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" event={"ID":"c3aebc7d-b6f0-4e4a-bd51-1373175105a8","Type":"ContainerStarted","Data":"3d6036e1d84aedac3d8f13a0849df3e868b0805f61f8be791da305f681e6458d"} Jul 07 15:07:45.139494 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:45.139449 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" Jul 07 15:07:45.139798 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:45.139745 2555 patch_prober.go:28] interesting pod/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 07 15:07:45.139880 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:45.139845 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" podUID="c3aebc7d-b6f0-4e4a-bd51-1373175105a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 07 15:07:46.139519 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:46.139482 2555 patch_prober.go:28] interesting pod/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 07 15:07:46.140131 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:46.140096 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" podUID="c3aebc7d-b6f0-4e4a-bd51-1373175105a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 07 15:07:47.139478 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:47.139431 2555 patch_prober.go:28] interesting pod/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 07 15:07:47.139698 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:47.139518 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" podUID="c3aebc7d-b6f0-4e4a-bd51-1373175105a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 07 15:07:48.139230 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:48.139183 2555 patch_prober.go:28] interesting pod/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 07 15:07:48.139463 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:48.139253 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" podUID="c3aebc7d-b6f0-4e4a-bd51-1373175105a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 07 15:07:49.142795 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:49.142762 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" Jul 07 15:07:49.143220 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:49.143026 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" Jul 07 15:07:49.143812 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:07:49.143792 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" Jul 07 15:08:01.846709 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:08:01.846674 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-09af33f3-5979f57f56-k2gg2"] Jul 07 15:08:01.847298 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:08:01.846924 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-09af33f3-5979f57f56-k2gg2" podUID="734daae0-3ff2-48fd-b2d2-d8eb05d08e63" containerName="maas-api" containerID="cri-o://de9e65b89d2bb50d8c40a4b9db2e1a4774129e742cc1db12ed4e074541f35367" gracePeriod=30 Jul 07 15:08:02.100891 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:08:02.100808 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-09af33f3-5979f57f56-k2gg2" Jul 07 15:08:02.222910 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:08:02.222872 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npsfm\" (UniqueName: \"kubernetes.io/projected/734daae0-3ff2-48fd-b2d2-d8eb05d08e63-kube-api-access-npsfm\") pod \"734daae0-3ff2-48fd-b2d2-d8eb05d08e63\" (UID: \"734daae0-3ff2-48fd-b2d2-d8eb05d08e63\") " Jul 07 15:08:02.222910 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:08:02.222908 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/734daae0-3ff2-48fd-b2d2-d8eb05d08e63-maas-api-tls\") pod \"734daae0-3ff2-48fd-b2d2-d8eb05d08e63\" (UID: \"734daae0-3ff2-48fd-b2d2-d8eb05d08e63\") " Jul 07 15:08:02.223130 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:08:02.222988 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/734daae0-3ff2-48fd-b2d2-d8eb05d08e63-openshift-service-ca\") pod \"734daae0-3ff2-48fd-b2d2-d8eb05d08e63\" (UID: \"734daae0-3ff2-48fd-b2d2-d8eb05d08e63\") " Jul 07 15:08:02.223377 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:08:02.223350 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/734daae0-3ff2-48fd-b2d2-d8eb05d08e63-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "734daae0-3ff2-48fd-b2d2-d8eb05d08e63" (UID: "734daae0-3ff2-48fd-b2d2-d8eb05d08e63"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:08:02.225178 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:08:02.225150 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/734daae0-3ff2-48fd-b2d2-d8eb05d08e63-kube-api-access-npsfm" (OuterVolumeSpecName: "kube-api-access-npsfm") pod "734daae0-3ff2-48fd-b2d2-d8eb05d08e63" (UID: "734daae0-3ff2-48fd-b2d2-d8eb05d08e63"). InnerVolumeSpecName "kube-api-access-npsfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:08:02.225178 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:08:02.225158 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/734daae0-3ff2-48fd-b2d2-d8eb05d08e63-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "734daae0-3ff2-48fd-b2d2-d8eb05d08e63" (UID: "734daae0-3ff2-48fd-b2d2-d8eb05d08e63"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:08:02.324112 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:08:02.324069 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/734daae0-3ff2-48fd-b2d2-d8eb05d08e63-openshift-service-ca\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:08:02.324112 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:08:02.324101 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npsfm\" (UniqueName: \"kubernetes.io/projected/734daae0-3ff2-48fd-b2d2-d8eb05d08e63-kube-api-access-npsfm\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:08:02.324112 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:08:02.324113 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/734daae0-3ff2-48fd-b2d2-d8eb05d08e63-maas-api-tls\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:08:02.646784 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:08:02.646722 2555 generic.go:356] "Generic (PLEG): container finished" podID="734daae0-3ff2-48fd-b2d2-d8eb05d08e63" containerID="de9e65b89d2bb50d8c40a4b9db2e1a4774129e742cc1db12ed4e074541f35367" exitCode=0 Jul 07 15:08:02.646987 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:08:02.646790 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-09af33f3-5979f57f56-k2gg2" Jul 07 15:08:02.646987 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:08:02.646788 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-09af33f3-5979f57f56-k2gg2" event={"ID":"734daae0-3ff2-48fd-b2d2-d8eb05d08e63","Type":"ContainerDied","Data":"de9e65b89d2bb50d8c40a4b9db2e1a4774129e742cc1db12ed4e074541f35367"} Jul 07 15:08:02.646987 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:08:02.646910 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-09af33f3-5979f57f56-k2gg2" event={"ID":"734daae0-3ff2-48fd-b2d2-d8eb05d08e63","Type":"ContainerDied","Data":"fd2b804bf4985d4555ba97bcf2d9a0a26358baeb11feeed456def6ad140d9952"} Jul 07 15:08:02.646987 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:08:02.646934 2555 scope.go:117] "RemoveContainer" containerID="de9e65b89d2bb50d8c40a4b9db2e1a4774129e742cc1db12ed4e074541f35367" Jul 07 15:08:02.656949 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:08:02.656931 2555 scope.go:117] "RemoveContainer" containerID="de9e65b89d2bb50d8c40a4b9db2e1a4774129e742cc1db12ed4e074541f35367" Jul 07 15:08:02.657270 ip-10-0-136-19 kubenswrapper[2555]: E0707 15:08:02.657252 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de9e65b89d2bb50d8c40a4b9db2e1a4774129e742cc1db12ed4e074541f35367\": container with ID starting with de9e65b89d2bb50d8c40a4b9db2e1a4774129e742cc1db12ed4e074541f35367 not found: ID does not exist" containerID="de9e65b89d2bb50d8c40a4b9db2e1a4774129e742cc1db12ed4e074541f35367" Jul 07 15:08:02.657315 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:08:02.657281 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de9e65b89d2bb50d8c40a4b9db2e1a4774129e742cc1db12ed4e074541f35367"} err="failed to get container status \"de9e65b89d2bb50d8c40a4b9db2e1a4774129e742cc1db12ed4e074541f35367\": rpc error: code = NotFound desc = could not find container \"de9e65b89d2bb50d8c40a4b9db2e1a4774129e742cc1db12ed4e074541f35367\": container with ID starting with de9e65b89d2bb50d8c40a4b9db2e1a4774129e742cc1db12ed4e074541f35367 not found: ID does not exist" Jul 07 15:08:02.671902 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:08:02.671860 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-09af33f3-5979f57f56-k2gg2"] Jul 07 15:08:02.674030 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:08:02.674004 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-09af33f3-5979f57f56-k2gg2"] Jul 07 15:08:03.043704 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:08:03.043588 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="734daae0-3ff2-48fd-b2d2-d8eb05d08e63" path="/var/lib/kubelet/pods/734daae0-3ff2-48fd-b2d2-d8eb05d08e63/volumes" Jul 07 15:09:04.261839 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:09:04.261799 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx"] Jul 07 15:09:04.262419 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:09:04.262175 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" podUID="c3aebc7d-b6f0-4e4a-bd51-1373175105a8" containerName="istio-proxy" containerID="cri-o://3d6036e1d84aedac3d8f13a0849df3e868b0805f61f8be791da305f681e6458d" gracePeriod=30 Jul 07 15:09:09.525612 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:09:09.525556 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" Jul 07 15:09:09.641034 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:09:09.641003 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-istio-envoy\") pod \"c3aebc7d-b6f0-4e4a-bd51-1373175105a8\" (UID: \"c3aebc7d-b6f0-4e4a-bd51-1373175105a8\") " Jul 07 15:09:09.641234 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:09:09.641054 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-istio-token\") pod \"c3aebc7d-b6f0-4e4a-bd51-1373175105a8\" (UID: \"c3aebc7d-b6f0-4e4a-bd51-1373175105a8\") " Jul 07 15:09:09.641234 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:09:09.641109 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-istio-podinfo\") pod \"c3aebc7d-b6f0-4e4a-bd51-1373175105a8\" (UID: \"c3aebc7d-b6f0-4e4a-bd51-1373175105a8\") " Jul 07 15:09:09.641234 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:09:09.641125 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-istiod-ca-cert\") pod \"c3aebc7d-b6f0-4e4a-bd51-1373175105a8\" (UID: \"c3aebc7d-b6f0-4e4a-bd51-1373175105a8\") " Jul 07 15:09:09.641234 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:09:09.641150 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pggn2\" (UniqueName: \"kubernetes.io/projected/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-kube-api-access-pggn2\") pod \"c3aebc7d-b6f0-4e4a-bd51-1373175105a8\" (UID: \"c3aebc7d-b6f0-4e4a-bd51-1373175105a8\") " Jul 07 15:09:09.641234 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:09:09.641200 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-credential-socket\") pod \"c3aebc7d-b6f0-4e4a-bd51-1373175105a8\" (UID: \"c3aebc7d-b6f0-4e4a-bd51-1373175105a8\") " Jul 07 15:09:09.641234 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:09:09.641216 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-workload-socket\") pod \"c3aebc7d-b6f0-4e4a-bd51-1373175105a8\" (UID: \"c3aebc7d-b6f0-4e4a-bd51-1373175105a8\") " Jul 07 15:09:09.641234 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:09:09.641232 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-istio-data\") pod \"c3aebc7d-b6f0-4e4a-bd51-1373175105a8\" (UID: \"c3aebc7d-b6f0-4e4a-bd51-1373175105a8\") " Jul 07 15:09:09.641667 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:09:09.641289 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-workload-certs\") pod \"c3aebc7d-b6f0-4e4a-bd51-1373175105a8\" (UID: \"c3aebc7d-b6f0-4e4a-bd51-1373175105a8\") " Jul 07 15:09:09.641667 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:09:09.641537 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c3aebc7d-b6f0-4e4a-bd51-1373175105a8" (UID: "c3aebc7d-b6f0-4e4a-bd51-1373175105a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:09:09.641667 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:09:09.641590 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c3aebc7d-b6f0-4e4a-bd51-1373175105a8" (UID: "c3aebc7d-b6f0-4e4a-bd51-1373175105a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:09:09.641667 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:09:09.641603 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c3aebc7d-b6f0-4e4a-bd51-1373175105a8" (UID: "c3aebc7d-b6f0-4e4a-bd51-1373175105a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:09:09.641915 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:09:09.641699 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c3aebc7d-b6f0-4e4a-bd51-1373175105a8" (UID: "c3aebc7d-b6f0-4e4a-bd51-1373175105a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:09:09.642303 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:09:09.642282 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-istio-data" (OuterVolumeSpecName: "istio-data") pod "c3aebc7d-b6f0-4e4a-bd51-1373175105a8" (UID: "c3aebc7d-b6f0-4e4a-bd51-1373175105a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:09:09.643375 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:09:09.643352 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c3aebc7d-b6f0-4e4a-bd51-1373175105a8" (UID: "c3aebc7d-b6f0-4e4a-bd51-1373175105a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 15:09:09.643497 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:09:09.643459 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c3aebc7d-b6f0-4e4a-bd51-1373175105a8" (UID: "c3aebc7d-b6f0-4e4a-bd51-1373175105a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:09:09.643788 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:09:09.643760 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-kube-api-access-pggn2" (OuterVolumeSpecName: "kube-api-access-pggn2") pod "c3aebc7d-b6f0-4e4a-bd51-1373175105a8" (UID: "c3aebc7d-b6f0-4e4a-bd51-1373175105a8"). InnerVolumeSpecName "kube-api-access-pggn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:09:09.643898 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:09:09.643783 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-istio-token" (OuterVolumeSpecName: "istio-token") pod "c3aebc7d-b6f0-4e4a-bd51-1373175105a8" (UID: "c3aebc7d-b6f0-4e4a-bd51-1373175105a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:09:09.742082 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:09:09.742049 2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-credential-socket\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:09:09.742082 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:09:09.742077 2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-workload-socket\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:09:09.742082 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:09:09.742090 2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-istio-data\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:09:09.742320 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:09:09.742102 2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-workload-certs\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:09:09.742320 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:09:09.742113 2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-istio-envoy\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:09:09.742320 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:09:09.742123 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-istio-token\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:09:09.742320 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:09:09.742134 2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-istio-podinfo\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:09:09.742320 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:09:09.742144 2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-istiod-ca-cert\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:09:09.742320 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:09:09.742157 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pggn2\" (UniqueName: \"kubernetes.io/projected/c3aebc7d-b6f0-4e4a-bd51-1373175105a8-kube-api-access-pggn2\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:09:09.973909 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:09:09.973867 2555 generic.go:356] "Generic (PLEG): container finished" podID="c3aebc7d-b6f0-4e4a-bd51-1373175105a8" containerID="3d6036e1d84aedac3d8f13a0849df3e868b0805f61f8be791da305f681e6458d" exitCode=0 Jul 07 15:09:09.974076 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:09:09.973922 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" Jul 07 15:09:09.974076 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:09:09.973941 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" event={"ID":"c3aebc7d-b6f0-4e4a-bd51-1373175105a8","Type":"ContainerDied","Data":"3d6036e1d84aedac3d8f13a0849df3e868b0805f61f8be791da305f681e6458d"} Jul 07 15:09:09.974076 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:09:09.973990 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx" event={"ID":"c3aebc7d-b6f0-4e4a-bd51-1373175105a8","Type":"ContainerDied","Data":"02946070b3e49107b59c43da7870db0266891835ddfc8b71db767b0fea2783f1"} Jul 07 15:09:09.974076 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:09:09.974010 2555 scope.go:117] "RemoveContainer" containerID="3d6036e1d84aedac3d8f13a0849df3e868b0805f61f8be791da305f681e6458d" Jul 07 15:09:09.984962 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:09:09.984927 2555 scope.go:117] "RemoveContainer" containerID="71edaefa0a005117e1444a2c7c609ec6541fe2e8c239ab1931312a40fd66a787" Jul 07 15:09:09.993457 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:09:09.993437 2555 scope.go:117] "RemoveContainer" containerID="3d6036e1d84aedac3d8f13a0849df3e868b0805f61f8be791da305f681e6458d" Jul 07 15:09:09.993731 ip-10-0-136-19 kubenswrapper[2555]: E0707 15:09:09.993711 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d6036e1d84aedac3d8f13a0849df3e868b0805f61f8be791da305f681e6458d\": container with ID starting with 3d6036e1d84aedac3d8f13a0849df3e868b0805f61f8be791da305f681e6458d not found: ID does not exist" containerID="3d6036e1d84aedac3d8f13a0849df3e868b0805f61f8be791da305f681e6458d" Jul 07 15:09:09.993836 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:09:09.993739 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d6036e1d84aedac3d8f13a0849df3e868b0805f61f8be791da305f681e6458d"} err="failed to get container status \"3d6036e1d84aedac3d8f13a0849df3e868b0805f61f8be791da305f681e6458d\": rpc error: code = NotFound desc = could not find container \"3d6036e1d84aedac3d8f13a0849df3e868b0805f61f8be791da305f681e6458d\": container with ID starting with 3d6036e1d84aedac3d8f13a0849df3e868b0805f61f8be791da305f681e6458d not found: ID does not exist" Jul 07 15:09:09.993898 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:09:09.993840 2555 scope.go:117] "RemoveContainer" containerID="71edaefa0a005117e1444a2c7c609ec6541fe2e8c239ab1931312a40fd66a787" Jul 07 15:09:09.994083 ip-10-0-136-19 kubenswrapper[2555]: E0707 15:09:09.994066 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71edaefa0a005117e1444a2c7c609ec6541fe2e8c239ab1931312a40fd66a787\": container with ID starting with 71edaefa0a005117e1444a2c7c609ec6541fe2e8c239ab1931312a40fd66a787 not found: ID does not exist" containerID="71edaefa0a005117e1444a2c7c609ec6541fe2e8c239ab1931312a40fd66a787" Jul 07 15:09:09.994141 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:09:09.994091 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71edaefa0a005117e1444a2c7c609ec6541fe2e8c239ab1931312a40fd66a787"} err="failed to get container status \"71edaefa0a005117e1444a2c7c609ec6541fe2e8c239ab1931312a40fd66a787\": rpc error: code = NotFound desc = could not find container \"71edaefa0a005117e1444a2c7c609ec6541fe2e8c239ab1931312a40fd66a787\": container with ID starting with 71edaefa0a005117e1444a2c7c609ec6541fe2e8c239ab1931312a40fd66a787 not found: ID does not exist" Jul 07 15:09:10.017165 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:09:10.017137 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx"] Jul 07 15:09:10.024845 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:09:10.024822 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-09af33f3-openshift-default-f9bb55f5f-hzgcx"] Jul 07 15:09:11.042229 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:09:11.042196 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3aebc7d-b6f0-4e4a-bd51-1373175105a8" path="/var/lib/kubelet/pods/c3aebc7d-b6f0-4e4a-bd51-1373175105a8/volumes" Jul 07 15:10:00.835078 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:00.835042 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff"] Jul 07 15:10:00.835825 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:00.835401 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3aebc7d-b6f0-4e4a-bd51-1373175105a8" containerName="istio-proxy" Jul 07 15:10:00.835825 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:00.835413 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3aebc7d-b6f0-4e4a-bd51-1373175105a8" containerName="istio-proxy" Jul 07 15:10:00.835825 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:00.835432 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3aebc7d-b6f0-4e4a-bd51-1373175105a8" containerName="istio-proxy" Jul 07 15:10:00.835825 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:00.835437 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3aebc7d-b6f0-4e4a-bd51-1373175105a8" containerName="istio-proxy" Jul 07 15:10:00.835825 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:00.835443 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="734daae0-3ff2-48fd-b2d2-d8eb05d08e63" containerName="maas-api" Jul 07 15:10:00.835825 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:00.835449 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="734daae0-3ff2-48fd-b2d2-d8eb05d08e63" containerName="maas-api" Jul 07 15:10:00.835825 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:00.835498 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3aebc7d-b6f0-4e4a-bd51-1373175105a8" containerName="istio-proxy" Jul 07 15:10:00.835825 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:00.835507 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="734daae0-3ff2-48fd-b2d2-d8eb05d08e63" containerName="maas-api" Jul 07 15:10:00.838445 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:00.838425 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" Jul 07 15:10:00.841564 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:00.841546 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-45adb8-openshift-default-dockercfg-f4c8n\"" Jul 07 15:10:00.852738 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:00.852719 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff"] Jul 07 15:10:00.901111 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:00.901082 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4tbk\" (UniqueName: \"kubernetes.io/projected/6dace71e-73b8-4977-a44f-6f5ef30661ff-kube-api-access-j4tbk\") pod \"e2e-inf-a-45adb8-openshift-default-56f787878f-45dff\" (UID: \"6dace71e-73b8-4977-a44f-6f5ef30661ff\") " pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" Jul 07 15:10:00.901250 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:00.901125 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dace71e-73b8-4977-a44f-6f5ef30661ff-workload-certs\") pod \"e2e-inf-a-45adb8-openshift-default-56f787878f-45dff\" (UID: \"6dace71e-73b8-4977-a44f-6f5ef30661ff\") " pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" Jul 07 15:10:00.901250 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:00.901153 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dace71e-73b8-4977-a44f-6f5ef30661ff-istio-token\") pod \"e2e-inf-a-45adb8-openshift-default-56f787878f-45dff\" (UID: \"6dace71e-73b8-4977-a44f-6f5ef30661ff\") " pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" Jul 07 15:10:00.901250 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:00.901179 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dace71e-73b8-4977-a44f-6f5ef30661ff-istio-envoy\") pod \"e2e-inf-a-45adb8-openshift-default-56f787878f-45dff\" (UID: \"6dace71e-73b8-4977-a44f-6f5ef30661ff\") " pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" Jul 07 15:10:00.901250 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:00.901207 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dace71e-73b8-4977-a44f-6f5ef30661ff-istio-podinfo\") pod \"e2e-inf-a-45adb8-openshift-default-56f787878f-45dff\" (UID: \"6dace71e-73b8-4977-a44f-6f5ef30661ff\") " pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" Jul 07 15:10:00.901250 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:00.901225 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dace71e-73b8-4977-a44f-6f5ef30661ff-istiod-ca-cert\") pod \"e2e-inf-a-45adb8-openshift-default-56f787878f-45dff\" (UID: \"6dace71e-73b8-4977-a44f-6f5ef30661ff\") " pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" Jul 07 15:10:00.901427 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:00.901279 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dace71e-73b8-4977-a44f-6f5ef30661ff-credential-socket\") pod \"e2e-inf-a-45adb8-openshift-default-56f787878f-45dff\" (UID: \"6dace71e-73b8-4977-a44f-6f5ef30661ff\") " pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" Jul 07 15:10:00.901427 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:00.901308 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dace71e-73b8-4977-a44f-6f5ef30661ff-istio-data\") pod \"e2e-inf-a-45adb8-openshift-default-56f787878f-45dff\" (UID: \"6dace71e-73b8-4977-a44f-6f5ef30661ff\") " pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" Jul 07 15:10:00.901427 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:00.901330 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dace71e-73b8-4977-a44f-6f5ef30661ff-workload-socket\") pod \"e2e-inf-a-45adb8-openshift-default-56f787878f-45dff\" (UID: \"6dace71e-73b8-4977-a44f-6f5ef30661ff\") " pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" Jul 07 15:10:01.002659 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:01.002621 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dace71e-73b8-4977-a44f-6f5ef30661ff-workload-certs\") pod \"e2e-inf-a-45adb8-openshift-default-56f787878f-45dff\" (UID: \"6dace71e-73b8-4977-a44f-6f5ef30661ff\") " pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" Jul 07 15:10:01.002886 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:01.002672 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dace71e-73b8-4977-a44f-6f5ef30661ff-istio-token\") pod \"e2e-inf-a-45adb8-openshift-default-56f787878f-45dff\" (UID: \"6dace71e-73b8-4977-a44f-6f5ef30661ff\") " pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" Jul 07 15:10:01.002886 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:01.002696 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dace71e-73b8-4977-a44f-6f5ef30661ff-istio-envoy\") pod \"e2e-inf-a-45adb8-openshift-default-56f787878f-45dff\" (UID: \"6dace71e-73b8-4977-a44f-6f5ef30661ff\") " pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" Jul 07 15:10:01.002886 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:01.002714 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dace71e-73b8-4977-a44f-6f5ef30661ff-istio-podinfo\") pod \"e2e-inf-a-45adb8-openshift-default-56f787878f-45dff\" (UID: \"6dace71e-73b8-4977-a44f-6f5ef30661ff\") " pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" Jul 07 15:10:01.002886 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:01.002733 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dace71e-73b8-4977-a44f-6f5ef30661ff-istiod-ca-cert\") pod \"e2e-inf-a-45adb8-openshift-default-56f787878f-45dff\" (UID: \"6dace71e-73b8-4977-a44f-6f5ef30661ff\") " pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" Jul 07 15:10:01.002886 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:01.002796 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dace71e-73b8-4977-a44f-6f5ef30661ff-credential-socket\") pod \"e2e-inf-a-45adb8-openshift-default-56f787878f-45dff\" (UID: \"6dace71e-73b8-4977-a44f-6f5ef30661ff\") " pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" Jul 07 15:10:01.003178 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:01.002930 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dace71e-73b8-4977-a44f-6f5ef30661ff-istio-data\") pod \"e2e-inf-a-45adb8-openshift-default-56f787878f-45dff\" (UID: \"6dace71e-73b8-4977-a44f-6f5ef30661ff\") " pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" Jul 07 15:10:01.003178 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:01.002983 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dace71e-73b8-4977-a44f-6f5ef30661ff-workload-socket\") pod \"e2e-inf-a-45adb8-openshift-default-56f787878f-45dff\" (UID: \"6dace71e-73b8-4977-a44f-6f5ef30661ff\") " pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" Jul 07 15:10:01.003178 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:01.003055 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4tbk\" (UniqueName: \"kubernetes.io/projected/6dace71e-73b8-4977-a44f-6f5ef30661ff-kube-api-access-j4tbk\") pod \"e2e-inf-a-45adb8-openshift-default-56f787878f-45dff\" (UID: \"6dace71e-73b8-4977-a44f-6f5ef30661ff\") " pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" Jul 07 15:10:01.003178 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:01.003097 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dace71e-73b8-4977-a44f-6f5ef30661ff-workload-certs\") pod \"e2e-inf-a-45adb8-openshift-default-56f787878f-45dff\" (UID: \"6dace71e-73b8-4977-a44f-6f5ef30661ff\") " pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" Jul 07 15:10:01.003397 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:01.003200 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dace71e-73b8-4977-a44f-6f5ef30661ff-credential-socket\") pod \"e2e-inf-a-45adb8-openshift-default-56f787878f-45dff\" (UID: \"6dace71e-73b8-4977-a44f-6f5ef30661ff\") " pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" Jul 07 15:10:01.003453 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:01.003415 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dace71e-73b8-4977-a44f-6f5ef30661ff-istio-data\") pod \"e2e-inf-a-45adb8-openshift-default-56f787878f-45dff\" (UID: \"6dace71e-73b8-4977-a44f-6f5ef30661ff\") " pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" Jul 07 15:10:01.003541 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:01.003518 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dace71e-73b8-4977-a44f-6f5ef30661ff-workload-socket\") pod \"e2e-inf-a-45adb8-openshift-default-56f787878f-45dff\" (UID: \"6dace71e-73b8-4977-a44f-6f5ef30661ff\") " pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" Jul 07 15:10:01.003609 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:01.003543 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dace71e-73b8-4977-a44f-6f5ef30661ff-istiod-ca-cert\") pod \"e2e-inf-a-45adb8-openshift-default-56f787878f-45dff\" (UID: \"6dace71e-73b8-4977-a44f-6f5ef30661ff\") " pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" Jul 07 15:10:01.005440 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:01.005420 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dace71e-73b8-4977-a44f-6f5ef30661ff-istio-podinfo\") pod \"e2e-inf-a-45adb8-openshift-default-56f787878f-45dff\" (UID: \"6dace71e-73b8-4977-a44f-6f5ef30661ff\") " pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" Jul 07 15:10:01.005440 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:01.005427 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dace71e-73b8-4977-a44f-6f5ef30661ff-istio-envoy\") pod \"e2e-inf-a-45adb8-openshift-default-56f787878f-45dff\" (UID: \"6dace71e-73b8-4977-a44f-6f5ef30661ff\") " pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" Jul 07 15:10:01.018403 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:01.018375 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4tbk\" (UniqueName: \"kubernetes.io/projected/6dace71e-73b8-4977-a44f-6f5ef30661ff-kube-api-access-j4tbk\") pod \"e2e-inf-a-45adb8-openshift-default-56f787878f-45dff\" (UID: \"6dace71e-73b8-4977-a44f-6f5ef30661ff\") " pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" Jul 07 15:10:01.025625 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:01.025597 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dace71e-73b8-4977-a44f-6f5ef30661ff-istio-token\") pod \"e2e-inf-a-45adb8-openshift-default-56f787878f-45dff\" (UID: \"6dace71e-73b8-4977-a44f-6f5ef30661ff\") " pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" Jul 07 15:10:01.148584 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:01.148497 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" Jul 07 15:10:01.297456 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:01.297429 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff"] Jul 07 15:10:01.298550 ip-10-0-136-19 kubenswrapper[2555]: W0707 15:10:01.298519 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dace71e_73b8_4977_a44f_6f5ef30661ff.slice/crio-e286114506edbd6a9cc6505ebff0b875f0290708f41d64949a1127d1cdcaf73d WatchSource:0}: Error finding container e286114506edbd6a9cc6505ebff0b875f0290708f41d64949a1127d1cdcaf73d: Status 404 returned error can't find the container with id e286114506edbd6a9cc6505ebff0b875f0290708f41d64949a1127d1cdcaf73d Jul 07 15:10:01.300533 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:01.300502 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:10:01.300642 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:01.300578 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:10:01.300642 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:01.300625 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:10:02.212972 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:02.212933 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" event={"ID":"6dace71e-73b8-4977-a44f-6f5ef30661ff","Type":"ContainerStarted","Data":"6e76985dec8c336cc9b625dd3a71b2fe3935081576d821385736fac12840c047"} Jul 07 15:10:02.213365 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:02.212998 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" event={"ID":"6dace71e-73b8-4977-a44f-6f5ef30661ff","Type":"ContainerStarted","Data":"e286114506edbd6a9cc6505ebff0b875f0290708f41d64949a1127d1cdcaf73d"} Jul 07 15:10:02.242576 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:02.242501 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" podStartSLOduration=2.24248373 podStartE2EDuration="2.24248373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:10:02.240786654 +0000 UTC m=+2143.664973128" watchObservedRunningTime="2026-07-07 15:10:02.24248373 +0000 UTC m=+2143.666670260" Jul 07 15:10:03.149053 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:03.149020 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" Jul 07 15:10:03.153919 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:03.153890 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" Jul 07 15:10:03.217928 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:03.217899 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" Jul 07 15:10:03.218956 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:03.218935 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" Jul 07 15:10:04.272038 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:04.272004 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-45adb8-5798c979c-2cmt4"] Jul 07 15:10:04.272706 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:04.272684 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3aebc7d-b6f0-4e4a-bd51-1373175105a8" containerName="istio-proxy" Jul 07 15:10:04.276118 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:04.276100 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-45adb8-5798c979c-2cmt4" Jul 07 15:10:04.278717 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:04.278694 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-45adb8-serving-cert\"" Jul 07 15:10:04.288527 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:04.288501 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-45adb8-5798c979c-2cmt4"] Jul 07 15:10:04.438945 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:04.438909 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74p77\" (UniqueName: \"kubernetes.io/projected/44cf7a97-38d9-4f56-b106-d43b3535edc8-kube-api-access-74p77\") pod \"maas-api-e2e-inf-a-45adb8-5798c979c-2cmt4\" (UID: \"44cf7a97-38d9-4f56-b106-d43b3535edc8\") " pod="opendatahub/maas-api-e2e-inf-a-45adb8-5798c979c-2cmt4" Jul 07 15:10:04.439147 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:04.438988 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44cf7a97-38d9-4f56-b106-d43b3535edc8-maas-api-tls\") pod \"maas-api-e2e-inf-a-45adb8-5798c979c-2cmt4\" (UID: \"44cf7a97-38d9-4f56-b106-d43b3535edc8\") " pod="opendatahub/maas-api-e2e-inf-a-45adb8-5798c979c-2cmt4" Jul 07 15:10:04.439147 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:04.439050 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44cf7a97-38d9-4f56-b106-d43b3535edc8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-45adb8-5798c979c-2cmt4\" (UID: \"44cf7a97-38d9-4f56-b106-d43b3535edc8\") " pod="opendatahub/maas-api-e2e-inf-a-45adb8-5798c979c-2cmt4" Jul 07 15:10:04.539950 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:04.539860 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44cf7a97-38d9-4f56-b106-d43b3535edc8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-45adb8-5798c979c-2cmt4\" (UID: \"44cf7a97-38d9-4f56-b106-d43b3535edc8\") " pod="opendatahub/maas-api-e2e-inf-a-45adb8-5798c979c-2cmt4" Jul 07 15:10:04.539950 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:04.539913 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74p77\" (UniqueName: \"kubernetes.io/projected/44cf7a97-38d9-4f56-b106-d43b3535edc8-kube-api-access-74p77\") pod \"maas-api-e2e-inf-a-45adb8-5798c979c-2cmt4\" (UID: \"44cf7a97-38d9-4f56-b106-d43b3535edc8\") " pod="opendatahub/maas-api-e2e-inf-a-45adb8-5798c979c-2cmt4" Jul 07 15:10:04.540158 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:04.539974 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44cf7a97-38d9-4f56-b106-d43b3535edc8-maas-api-tls\") pod \"maas-api-e2e-inf-a-45adb8-5798c979c-2cmt4\" (UID: \"44cf7a97-38d9-4f56-b106-d43b3535edc8\") " pod="opendatahub/maas-api-e2e-inf-a-45adb8-5798c979c-2cmt4" Jul 07 15:10:04.540158 ip-10-0-136-19 kubenswrapper[2555]: E0707 15:10:04.540084 2555 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-a-45adb8-serving-cert: secret "maas-api-e2e-inf-a-45adb8-serving-cert" not found Jul 07 15:10:04.540158 ip-10-0-136-19 kubenswrapper[2555]: E0707 15:10:04.540153 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44cf7a97-38d9-4f56-b106-d43b3535edc8-maas-api-tls podName:44cf7a97-38d9-4f56-b106-d43b3535edc8 nodeName:}" failed. No retries permitted until 2026-07-07 15:10:05.040135448 +0000 UTC m=+2146.464321901 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/44cf7a97-38d9-4f56-b106-d43b3535edc8-maas-api-tls") pod "maas-api-e2e-inf-a-45adb8-5798c979c-2cmt4" (UID: "44cf7a97-38d9-4f56-b106-d43b3535edc8") : secret "maas-api-e2e-inf-a-45adb8-serving-cert" not found Jul 07 15:10:04.540488 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:04.540467 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44cf7a97-38d9-4f56-b106-d43b3535edc8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-45adb8-5798c979c-2cmt4\" (UID: \"44cf7a97-38d9-4f56-b106-d43b3535edc8\") " pod="opendatahub/maas-api-e2e-inf-a-45adb8-5798c979c-2cmt4" Jul 07 15:10:04.553382 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:04.553353 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74p77\" (UniqueName: \"kubernetes.io/projected/44cf7a97-38d9-4f56-b106-d43b3535edc8-kube-api-access-74p77\") pod \"maas-api-e2e-inf-a-45adb8-5798c979c-2cmt4\" (UID: \"44cf7a97-38d9-4f56-b106-d43b3535edc8\") " pod="opendatahub/maas-api-e2e-inf-a-45adb8-5798c979c-2cmt4" Jul 07 15:10:05.044235 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:05.044207 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44cf7a97-38d9-4f56-b106-d43b3535edc8-maas-api-tls\") pod \"maas-api-e2e-inf-a-45adb8-5798c979c-2cmt4\" (UID: \"44cf7a97-38d9-4f56-b106-d43b3535edc8\") " pod="opendatahub/maas-api-e2e-inf-a-45adb8-5798c979c-2cmt4" Jul 07 15:10:05.046511 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:05.046492 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44cf7a97-38d9-4f56-b106-d43b3535edc8-maas-api-tls\") pod \"maas-api-e2e-inf-a-45adb8-5798c979c-2cmt4\" (UID: \"44cf7a97-38d9-4f56-b106-d43b3535edc8\") " pod="opendatahub/maas-api-e2e-inf-a-45adb8-5798c979c-2cmt4" Jul 07 15:10:05.187741 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:05.187705 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-45adb8-5798c979c-2cmt4" Jul 07 15:10:05.332206 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:05.332171 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-45adb8-5798c979c-2cmt4"] Jul 07 15:10:05.335579 ip-10-0-136-19 kubenswrapper[2555]: W0707 15:10:05.335554 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44cf7a97_38d9_4f56_b106_d43b3535edc8.slice/crio-a661c49cba37292cea8bb0ad37665bd8d42c2db066e0616871a6749797c39e82 WatchSource:0}: Error finding container a661c49cba37292cea8bb0ad37665bd8d42c2db066e0616871a6749797c39e82: Status 404 returned error can't find the container with id a661c49cba37292cea8bb0ad37665bd8d42c2db066e0616871a6749797c39e82 Jul 07 15:10:05.337162 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:05.337145 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 15:10:06.234127 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:06.234034 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-45adb8-5798c979c-2cmt4" event={"ID":"44cf7a97-38d9-4f56-b106-d43b3535edc8","Type":"ContainerStarted","Data":"141dff5b5cb30b585c6eb91d094daf490fba3ff25dce40cd99b81529f08857aa"} Jul 07 15:10:06.234127 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:06.234089 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-45adb8-5798c979c-2cmt4" event={"ID":"44cf7a97-38d9-4f56-b106-d43b3535edc8","Type":"ContainerStarted","Data":"a661c49cba37292cea8bb0ad37665bd8d42c2db066e0616871a6749797c39e82"} Jul 07 15:10:06.234321 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:06.234129 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-45adb8-5798c979c-2cmt4" Jul 07 15:10:06.256204 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:06.256150 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-45adb8-5798c979c-2cmt4" podStartSLOduration=1.677065637 podStartE2EDuration="2.256128806s" totalImagesPullingTime="579.063169ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:10:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:10:06.254432373 +0000 UTC m=+2147.678618850" watchObservedRunningTime="2026-07-07 15:10:06.256128806 +0000 UTC m=+2147.680315281" Jul 07 15:10:12.123800 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:12.123712 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-b-c95db0-5f94fb5cd9-rj7sp"] Jul 07 15:10:12.127347 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:12.127320 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-c95db0-5f94fb5cd9-rj7sp" Jul 07 15:10:12.130804 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:12.130785 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-b-c95db0-serving-cert\"" Jul 07 15:10:12.142722 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:12.142694 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-c95db0-5f94fb5cd9-rj7sp"] Jul 07 15:10:12.244795 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:12.244764 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-45adb8-5798c979c-2cmt4" Jul 07 15:10:12.311072 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:12.311043 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79f774fc-b54f-49e9-ae60-1fdf5ca78c89-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c95db0-5f94fb5cd9-rj7sp\" (UID: \"79f774fc-b54f-49e9-ae60-1fdf5ca78c89\") " pod="opendatahub/maas-api-e2e-inf-b-c95db0-5f94fb5cd9-rj7sp" Jul 07 15:10:12.311072 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:12.311079 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbhwf\" (UniqueName: \"kubernetes.io/projected/79f774fc-b54f-49e9-ae60-1fdf5ca78c89-kube-api-access-nbhwf\") pod \"maas-api-e2e-inf-b-c95db0-5f94fb5cd9-rj7sp\" (UID: \"79f774fc-b54f-49e9-ae60-1fdf5ca78c89\") " pod="opendatahub/maas-api-e2e-inf-b-c95db0-5f94fb5cd9-rj7sp" Jul 07 15:10:12.311297 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:12.311233 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79f774fc-b54f-49e9-ae60-1fdf5ca78c89-maas-api-tls\") pod \"maas-api-e2e-inf-b-c95db0-5f94fb5cd9-rj7sp\" (UID: \"79f774fc-b54f-49e9-ae60-1fdf5ca78c89\") " pod="opendatahub/maas-api-e2e-inf-b-c95db0-5f94fb5cd9-rj7sp" Jul 07 15:10:12.412514 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:12.412413 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79f774fc-b54f-49e9-ae60-1fdf5ca78c89-maas-api-tls\") pod \"maas-api-e2e-inf-b-c95db0-5f94fb5cd9-rj7sp\" (UID: \"79f774fc-b54f-49e9-ae60-1fdf5ca78c89\") " pod="opendatahub/maas-api-e2e-inf-b-c95db0-5f94fb5cd9-rj7sp" Jul 07 15:10:12.412514 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:12.412490 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79f774fc-b54f-49e9-ae60-1fdf5ca78c89-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c95db0-5f94fb5cd9-rj7sp\" (UID: \"79f774fc-b54f-49e9-ae60-1fdf5ca78c89\") " pod="opendatahub/maas-api-e2e-inf-b-c95db0-5f94fb5cd9-rj7sp" Jul 07 15:10:12.412803 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:12.412526 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbhwf\" (UniqueName: \"kubernetes.io/projected/79f774fc-b54f-49e9-ae60-1fdf5ca78c89-kube-api-access-nbhwf\") pod \"maas-api-e2e-inf-b-c95db0-5f94fb5cd9-rj7sp\" (UID: \"79f774fc-b54f-49e9-ae60-1fdf5ca78c89\") " pod="opendatahub/maas-api-e2e-inf-b-c95db0-5f94fb5cd9-rj7sp" Jul 07 15:10:12.412803 ip-10-0-136-19 kubenswrapper[2555]: E0707 15:10:12.412571 2555 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-b-c95db0-serving-cert: secret "maas-api-e2e-inf-b-c95db0-serving-cert" not found Jul 07 15:10:12.412803 ip-10-0-136-19 kubenswrapper[2555]: E0707 15:10:12.412633 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79f774fc-b54f-49e9-ae60-1fdf5ca78c89-maas-api-tls podName:79f774fc-b54f-49e9-ae60-1fdf5ca78c89 nodeName:}" failed. No retries permitted until 2026-07-07 15:10:12.912617106 +0000 UTC m=+2154.336803558 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/79f774fc-b54f-49e9-ae60-1fdf5ca78c89-maas-api-tls") pod "maas-api-e2e-inf-b-c95db0-5f94fb5cd9-rj7sp" (UID: "79f774fc-b54f-49e9-ae60-1fdf5ca78c89") : secret "maas-api-e2e-inf-b-c95db0-serving-cert" not found Jul 07 15:10:12.413186 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:12.413165 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79f774fc-b54f-49e9-ae60-1fdf5ca78c89-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c95db0-5f94fb5cd9-rj7sp\" (UID: \"79f774fc-b54f-49e9-ae60-1fdf5ca78c89\") " pod="opendatahub/maas-api-e2e-inf-b-c95db0-5f94fb5cd9-rj7sp" Jul 07 15:10:12.436112 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:12.436086 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbhwf\" (UniqueName: \"kubernetes.io/projected/79f774fc-b54f-49e9-ae60-1fdf5ca78c89-kube-api-access-nbhwf\") pod \"maas-api-e2e-inf-b-c95db0-5f94fb5cd9-rj7sp\" (UID: \"79f774fc-b54f-49e9-ae60-1fdf5ca78c89\") " pod="opendatahub/maas-api-e2e-inf-b-c95db0-5f94fb5cd9-rj7sp" Jul 07 15:10:12.915994 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:12.915961 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79f774fc-b54f-49e9-ae60-1fdf5ca78c89-maas-api-tls\") pod \"maas-api-e2e-inf-b-c95db0-5f94fb5cd9-rj7sp\" (UID: \"79f774fc-b54f-49e9-ae60-1fdf5ca78c89\") " pod="opendatahub/maas-api-e2e-inf-b-c95db0-5f94fb5cd9-rj7sp" Jul 07 15:10:12.918367 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:12.918344 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79f774fc-b54f-49e9-ae60-1fdf5ca78c89-maas-api-tls\") pod \"maas-api-e2e-inf-b-c95db0-5f94fb5cd9-rj7sp\" (UID: \"79f774fc-b54f-49e9-ae60-1fdf5ca78c89\") " pod="opendatahub/maas-api-e2e-inf-b-c95db0-5f94fb5cd9-rj7sp" Jul 07 15:10:13.038911 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:13.038858 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-c95db0-5f94fb5cd9-rj7sp" Jul 07 15:10:13.219009 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:13.218982 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-c95db0-5f94fb5cd9-rj7sp"] Jul 07 15:10:13.220838 ip-10-0-136-19 kubenswrapper[2555]: W0707 15:10:13.220811 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79f774fc_b54f_49e9_ae60_1fdf5ca78c89.slice/crio-6764d60164e577bd9224e3f38a554d0dbbf1062bea454a4eb1c970757829dd7b WatchSource:0}: Error finding container 6764d60164e577bd9224e3f38a554d0dbbf1062bea454a4eb1c970757829dd7b: Status 404 returned error can't find the container with id 6764d60164e577bd9224e3f38a554d0dbbf1062bea454a4eb1c970757829dd7b Jul 07 15:10:13.269222 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:13.269178 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-c95db0-5f94fb5cd9-rj7sp" event={"ID":"79f774fc-b54f-49e9-ae60-1fdf5ca78c89","Type":"ContainerStarted","Data":"6764d60164e577bd9224e3f38a554d0dbbf1062bea454a4eb1c970757829dd7b"} Jul 07 15:10:14.275954 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:14.275912 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-c95db0-5f94fb5cd9-rj7sp" event={"ID":"79f774fc-b54f-49e9-ae60-1fdf5ca78c89","Type":"ContainerStarted","Data":"9bfc45df0b16b865187c2265e02bf3a020ae26d29789dea0baaab1737febffcf"} Jul 07 15:10:14.276384 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:14.275980 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-b-c95db0-5f94fb5cd9-rj7sp" Jul 07 15:10:14.295547 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:14.295486 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-b-c95db0-5f94fb5cd9-rj7sp" podStartSLOduration=1.807148728 podStartE2EDuration="2.295463815s" totalImagesPullingTime="488.315087ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:10:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:10:14.292125652 +0000 UTC m=+2155.716312129" watchObservedRunningTime="2026-07-07 15:10:14.295463815 +0000 UTC m=+2155.719650293" Jul 07 15:10:19.258967 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:19.258918 2555 patch_prober.go:28] interesting pod/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 07 15:10:19.259487 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:19.258994 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" podUID="6dace71e-73b8-4977-a44f-6f5ef30661ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 07 15:10:20.286746 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:20.286711 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-b-c95db0-5f94fb5cd9-rj7sp" Jul 07 15:10:20.310103 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:20.310069 2555 generic.go:356] "Generic (PLEG): container finished" podID="6dace71e-73b8-4977-a44f-6f5ef30661ff" containerID="6e76985dec8c336cc9b625dd3a71b2fe3935081576d821385736fac12840c047" exitCode=0 Jul 07 15:10:20.310309 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:20.310143 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" event={"ID":"6dace71e-73b8-4977-a44f-6f5ef30661ff","Type":"ContainerDied","Data":"6e76985dec8c336cc9b625dd3a71b2fe3935081576d821385736fac12840c047"} Jul 07 15:10:20.310613 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:20.310595 2555 scope.go:117] "RemoveContainer" containerID="6e76985dec8c336cc9b625dd3a71b2fe3935081576d821385736fac12840c047" Jul 07 15:10:20.311794 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:20.311742 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:10:20.311863 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:20.311845 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:10:20.311910 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:20.311882 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:10:21.316368 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:21.316330 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" event={"ID":"6dace71e-73b8-4977-a44f-6f5ef30661ff","Type":"ContainerStarted","Data":"f4cdc55f6b9162f1a28ac841e515403fc23b1c1bd641b2fe0772d6dabc9de560"} Jul 07 15:10:22.149874 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:22.149811 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" Jul 07 15:10:22.150107 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:22.149898 2555 patch_prober.go:28] interesting pod/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 07 15:10:22.150107 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:22.149961 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" podUID="6dace71e-73b8-4977-a44f-6f5ef30661ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 07 15:10:23.150051 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:23.150005 2555 patch_prober.go:28] interesting pod/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 07 15:10:23.150706 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:23.150672 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" podUID="6dace71e-73b8-4977-a44f-6f5ef30661ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 07 15:10:24.149593 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:24.149544 2555 patch_prober.go:28] interesting pod/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 07 15:10:24.149837 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:24.149616 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" podUID="6dace71e-73b8-4977-a44f-6f5ef30661ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 07 15:10:25.149243 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:25.149190 2555 patch_prober.go:28] interesting pod/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 07 15:10:25.149829 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:25.149281 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" podUID="6dace71e-73b8-4977-a44f-6f5ef30661ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 07 15:10:26.154378 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:26.154340 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" Jul 07 15:10:26.154890 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:26.154862 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" Jul 07 15:10:26.156010 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:10:26.155983 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" Jul 07 15:11:11.585945 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:11.585845 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-45adb8-5798c979c-2cmt4"] Jul 07 15:11:11.586565 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:11.586097 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-45adb8-5798c979c-2cmt4" podUID="44cf7a97-38d9-4f56-b106-d43b3535edc8" containerName="maas-api" containerID="cri-o://141dff5b5cb30b585c6eb91d094daf490fba3ff25dce40cd99b81529f08857aa" gracePeriod=30 Jul 07 15:11:11.837418 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:11.837353 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-45adb8-5798c979c-2cmt4" Jul 07 15:11:11.983537 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:11.983499 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74p77\" (UniqueName: \"kubernetes.io/projected/44cf7a97-38d9-4f56-b106-d43b3535edc8-kube-api-access-74p77\") pod \"44cf7a97-38d9-4f56-b106-d43b3535edc8\" (UID: \"44cf7a97-38d9-4f56-b106-d43b3535edc8\") " Jul 07 15:11:11.983711 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:11.983575 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44cf7a97-38d9-4f56-b106-d43b3535edc8-openshift-service-ca\") pod \"44cf7a97-38d9-4f56-b106-d43b3535edc8\" (UID: \"44cf7a97-38d9-4f56-b106-d43b3535edc8\") " Jul 07 15:11:11.983711 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:11.983603 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44cf7a97-38d9-4f56-b106-d43b3535edc8-maas-api-tls\") pod \"44cf7a97-38d9-4f56-b106-d43b3535edc8\" (UID: \"44cf7a97-38d9-4f56-b106-d43b3535edc8\") " Jul 07 15:11:11.983996 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:11.983964 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44cf7a97-38d9-4f56-b106-d43b3535edc8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "44cf7a97-38d9-4f56-b106-d43b3535edc8" (UID: "44cf7a97-38d9-4f56-b106-d43b3535edc8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:11:11.985718 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:11.985693 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44cf7a97-38d9-4f56-b106-d43b3535edc8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "44cf7a97-38d9-4f56-b106-d43b3535edc8" (UID: "44cf7a97-38d9-4f56-b106-d43b3535edc8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:11:11.985718 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:11.985708 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44cf7a97-38d9-4f56-b106-d43b3535edc8-kube-api-access-74p77" (OuterVolumeSpecName: "kube-api-access-74p77") pod "44cf7a97-38d9-4f56-b106-d43b3535edc8" (UID: "44cf7a97-38d9-4f56-b106-d43b3535edc8"). InnerVolumeSpecName "kube-api-access-74p77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:11:12.085356 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:12.085314 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74p77\" (UniqueName: \"kubernetes.io/projected/44cf7a97-38d9-4f56-b106-d43b3535edc8-kube-api-access-74p77\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:11:12.085356 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:12.085348 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44cf7a97-38d9-4f56-b106-d43b3535edc8-openshift-service-ca\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:11:12.085356 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:12.085359 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44cf7a97-38d9-4f56-b106-d43b3535edc8-maas-api-tls\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:11:12.564402 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:12.564361 2555 generic.go:356] "Generic (PLEG): container finished" podID="44cf7a97-38d9-4f56-b106-d43b3535edc8" containerID="141dff5b5cb30b585c6eb91d094daf490fba3ff25dce40cd99b81529f08857aa" exitCode=0 Jul 07 15:11:12.564402 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:12.564409 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-45adb8-5798c979c-2cmt4" Jul 07 15:11:12.564619 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:12.564443 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-45adb8-5798c979c-2cmt4" event={"ID":"44cf7a97-38d9-4f56-b106-d43b3535edc8","Type":"ContainerDied","Data":"141dff5b5cb30b585c6eb91d094daf490fba3ff25dce40cd99b81529f08857aa"} Jul 07 15:11:12.564619 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:12.564501 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-45adb8-5798c979c-2cmt4" event={"ID":"44cf7a97-38d9-4f56-b106-d43b3535edc8","Type":"ContainerDied","Data":"a661c49cba37292cea8bb0ad37665bd8d42c2db066e0616871a6749797c39e82"} Jul 07 15:11:12.564619 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:12.564522 2555 scope.go:117] "RemoveContainer" containerID="141dff5b5cb30b585c6eb91d094daf490fba3ff25dce40cd99b81529f08857aa" Jul 07 15:11:12.573398 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:12.573380 2555 scope.go:117] "RemoveContainer" containerID="141dff5b5cb30b585c6eb91d094daf490fba3ff25dce40cd99b81529f08857aa" Jul 07 15:11:12.573684 ip-10-0-136-19 kubenswrapper[2555]: E0707 15:11:12.573666 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"141dff5b5cb30b585c6eb91d094daf490fba3ff25dce40cd99b81529f08857aa\": container with ID starting with 141dff5b5cb30b585c6eb91d094daf490fba3ff25dce40cd99b81529f08857aa not found: ID does not exist" containerID="141dff5b5cb30b585c6eb91d094daf490fba3ff25dce40cd99b81529f08857aa" Jul 07 15:11:12.573741 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:12.573692 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"141dff5b5cb30b585c6eb91d094daf490fba3ff25dce40cd99b81529f08857aa"} err="failed to get container status \"141dff5b5cb30b585c6eb91d094daf490fba3ff25dce40cd99b81529f08857aa\": rpc error: code = NotFound desc = could not find container \"141dff5b5cb30b585c6eb91d094daf490fba3ff25dce40cd99b81529f08857aa\": container with ID starting with 141dff5b5cb30b585c6eb91d094daf490fba3ff25dce40cd99b81529f08857aa not found: ID does not exist" Jul 07 15:11:12.588362 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:12.588332 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-45adb8-5798c979c-2cmt4"] Jul 07 15:11:12.591247 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:12.591224 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-45adb8-5798c979c-2cmt4"] Jul 07 15:11:13.041974 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:13.041944 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44cf7a97-38d9-4f56-b106-d43b3535edc8" path="/var/lib/kubelet/pods/44cf7a97-38d9-4f56-b106-d43b3535edc8/volumes" Jul 07 15:11:19.167296 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:19.167262 2555 scope.go:117] "RemoveContainer" containerID="b38ff5447abeb21429abfa4ea68c9ba623156c0e070b6209b9b964f2cd229880" Jul 07 15:11:54.214817 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:54.214701 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff"] Jul 07 15:11:54.215410 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:54.214979 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" podUID="6dace71e-73b8-4977-a44f-6f5ef30661ff" containerName="istio-proxy" containerID="cri-o://f4cdc55f6b9162f1a28ac841e515403fc23b1c1bd641b2fe0772d6dabc9de560" gracePeriod=30 Jul 07 15:11:55.437369 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:55.437326 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-c95db0-5f94fb5cd9-rj7sp"] Jul 07 15:11:55.437898 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:55.437539 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-b-c95db0-5f94fb5cd9-rj7sp" podUID="79f774fc-b54f-49e9-ae60-1fdf5ca78c89" containerName="maas-api" containerID="cri-o://9bfc45df0b16b865187c2265e02bf3a020ae26d29789dea0baaab1737febffcf" gracePeriod=30 Jul 07 15:11:55.688294 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:55.688218 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-c95db0-5f94fb5cd9-rj7sp" Jul 07 15:11:55.776274 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:55.776235 2555 generic.go:356] "Generic (PLEG): container finished" podID="79f774fc-b54f-49e9-ae60-1fdf5ca78c89" containerID="9bfc45df0b16b865187c2265e02bf3a020ae26d29789dea0baaab1737febffcf" exitCode=0 Jul 07 15:11:55.776448 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:55.776288 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-c95db0-5f94fb5cd9-rj7sp" Jul 07 15:11:55.776448 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:55.776323 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-c95db0-5f94fb5cd9-rj7sp" event={"ID":"79f774fc-b54f-49e9-ae60-1fdf5ca78c89","Type":"ContainerDied","Data":"9bfc45df0b16b865187c2265e02bf3a020ae26d29789dea0baaab1737febffcf"} Jul 07 15:11:55.776448 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:55.776410 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-c95db0-5f94fb5cd9-rj7sp" event={"ID":"79f774fc-b54f-49e9-ae60-1fdf5ca78c89","Type":"ContainerDied","Data":"6764d60164e577bd9224e3f38a554d0dbbf1062bea454a4eb1c970757829dd7b"} Jul 07 15:11:55.776448 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:55.776438 2555 scope.go:117] "RemoveContainer" containerID="9bfc45df0b16b865187c2265e02bf3a020ae26d29789dea0baaab1737febffcf" Jul 07 15:11:55.785267 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:55.785249 2555 scope.go:117] "RemoveContainer" containerID="9bfc45df0b16b865187c2265e02bf3a020ae26d29789dea0baaab1737febffcf" Jul 07 15:11:55.785589 ip-10-0-136-19 kubenswrapper[2555]: E0707 15:11:55.785568 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bfc45df0b16b865187c2265e02bf3a020ae26d29789dea0baaab1737febffcf\": container with ID starting with 9bfc45df0b16b865187c2265e02bf3a020ae26d29789dea0baaab1737febffcf not found: ID does not exist" containerID="9bfc45df0b16b865187c2265e02bf3a020ae26d29789dea0baaab1737febffcf" Jul 07 15:11:55.785638 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:55.785601 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bfc45df0b16b865187c2265e02bf3a020ae26d29789dea0baaab1737febffcf"} err="failed to get container status \"9bfc45df0b16b865187c2265e02bf3a020ae26d29789dea0baaab1737febffcf\": rpc error: code = NotFound desc = could not find container \"9bfc45df0b16b865187c2265e02bf3a020ae26d29789dea0baaab1737febffcf\": container with ID starting with 9bfc45df0b16b865187c2265e02bf3a020ae26d29789dea0baaab1737febffcf not found: ID does not exist" Jul 07 15:11:55.792857 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:55.792830 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79f774fc-b54f-49e9-ae60-1fdf5ca78c89-maas-api-tls\") pod \"79f774fc-b54f-49e9-ae60-1fdf5ca78c89\" (UID: \"79f774fc-b54f-49e9-ae60-1fdf5ca78c89\") " Jul 07 15:11:55.792940 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:55.792911 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79f774fc-b54f-49e9-ae60-1fdf5ca78c89-openshift-service-ca\") pod \"79f774fc-b54f-49e9-ae60-1fdf5ca78c89\" (UID: \"79f774fc-b54f-49e9-ae60-1fdf5ca78c89\") " Jul 07 15:11:55.792995 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:55.792984 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbhwf\" (UniqueName: \"kubernetes.io/projected/79f774fc-b54f-49e9-ae60-1fdf5ca78c89-kube-api-access-nbhwf\") pod \"79f774fc-b54f-49e9-ae60-1fdf5ca78c89\" (UID: \"79f774fc-b54f-49e9-ae60-1fdf5ca78c89\") " Jul 07 15:11:55.793304 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:55.793275 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79f774fc-b54f-49e9-ae60-1fdf5ca78c89-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "79f774fc-b54f-49e9-ae60-1fdf5ca78c89" (UID: "79f774fc-b54f-49e9-ae60-1fdf5ca78c89"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:11:55.795076 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:55.795048 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79f774fc-b54f-49e9-ae60-1fdf5ca78c89-kube-api-access-nbhwf" (OuterVolumeSpecName: "kube-api-access-nbhwf") pod "79f774fc-b54f-49e9-ae60-1fdf5ca78c89" (UID: "79f774fc-b54f-49e9-ae60-1fdf5ca78c89"). InnerVolumeSpecName "kube-api-access-nbhwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:11:55.795076 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:55.795051 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79f774fc-b54f-49e9-ae60-1fdf5ca78c89-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "79f774fc-b54f-49e9-ae60-1fdf5ca78c89" (UID: "79f774fc-b54f-49e9-ae60-1fdf5ca78c89"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:11:55.894237 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:55.894195 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79f774fc-b54f-49e9-ae60-1fdf5ca78c89-maas-api-tls\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:11:55.894237 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:55.894230 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79f774fc-b54f-49e9-ae60-1fdf5ca78c89-openshift-service-ca\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:11:55.894237 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:55.894245 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbhwf\" (UniqueName: \"kubernetes.io/projected/79f774fc-b54f-49e9-ae60-1fdf5ca78c89-kube-api-access-nbhwf\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:11:56.101900 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:56.101852 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-c95db0-5f94fb5cd9-rj7sp"] Jul 07 15:11:56.109803 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:56.109771 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-c95db0-5f94fb5cd9-rj7sp"] Jul 07 15:11:56.157590 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:56.157554 2555 patch_prober.go:28] interesting pod/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 07 15:11:56.157786 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:56.157610 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" podUID="6dace71e-73b8-4977-a44f-6f5ef30661ff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 15:11:57.042503 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:57.042465 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79f774fc-b54f-49e9-ae60-1fdf5ca78c89" path="/var/lib/kubelet/pods/79f774fc-b54f-49e9-ae60-1fdf5ca78c89/volumes" Jul 07 15:11:59.477090 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:59.477014 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" Jul 07 15:11:59.628945 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:59.628907 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dace71e-73b8-4977-a44f-6f5ef30661ff-istio-envoy\") pod \"6dace71e-73b8-4977-a44f-6f5ef30661ff\" (UID: \"6dace71e-73b8-4977-a44f-6f5ef30661ff\") " Jul 07 15:11:59.629142 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:59.628964 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dace71e-73b8-4977-a44f-6f5ef30661ff-istio-data\") pod \"6dace71e-73b8-4977-a44f-6f5ef30661ff\" (UID: \"6dace71e-73b8-4977-a44f-6f5ef30661ff\") " Jul 07 15:11:59.629142 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:59.628993 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dace71e-73b8-4977-a44f-6f5ef30661ff-workload-certs\") pod \"6dace71e-73b8-4977-a44f-6f5ef30661ff\" (UID: \"6dace71e-73b8-4977-a44f-6f5ef30661ff\") " Jul 07 15:11:59.629142 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:59.629051 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dace71e-73b8-4977-a44f-6f5ef30661ff-credential-socket\") pod \"6dace71e-73b8-4977-a44f-6f5ef30661ff\" (UID: \"6dace71e-73b8-4977-a44f-6f5ef30661ff\") " Jul 07 15:11:59.629142 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:59.629086 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4tbk\" (UniqueName: \"kubernetes.io/projected/6dace71e-73b8-4977-a44f-6f5ef30661ff-kube-api-access-j4tbk\") pod \"6dace71e-73b8-4977-a44f-6f5ef30661ff\" (UID: \"6dace71e-73b8-4977-a44f-6f5ef30661ff\") " Jul 07 15:11:59.629142 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:59.629112 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dace71e-73b8-4977-a44f-6f5ef30661ff-istio-token\") pod \"6dace71e-73b8-4977-a44f-6f5ef30661ff\" (UID: \"6dace71e-73b8-4977-a44f-6f5ef30661ff\") " Jul 07 15:11:59.629419 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:59.629184 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dace71e-73b8-4977-a44f-6f5ef30661ff-istio-podinfo\") pod \"6dace71e-73b8-4977-a44f-6f5ef30661ff\" (UID: \"6dace71e-73b8-4977-a44f-6f5ef30661ff\") " Jul 07 15:11:59.629419 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:59.629221 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dace71e-73b8-4977-a44f-6f5ef30661ff-istiod-ca-cert\") pod \"6dace71e-73b8-4977-a44f-6f5ef30661ff\" (UID: \"6dace71e-73b8-4977-a44f-6f5ef30661ff\") " Jul 07 15:11:59.629419 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:59.629266 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dace71e-73b8-4977-a44f-6f5ef30661ff-workload-socket\") pod \"6dace71e-73b8-4977-a44f-6f5ef30661ff\" (UID: \"6dace71e-73b8-4977-a44f-6f5ef30661ff\") " Jul 07 15:11:59.629419 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:59.629335 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dace71e-73b8-4977-a44f-6f5ef30661ff-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6dace71e-73b8-4977-a44f-6f5ef30661ff" (UID: "6dace71e-73b8-4977-a44f-6f5ef30661ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:11:59.629615 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:59.629569 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dace71e-73b8-4977-a44f-6f5ef30661ff-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6dace71e-73b8-4977-a44f-6f5ef30661ff" (UID: "6dace71e-73b8-4977-a44f-6f5ef30661ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:11:59.629666 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:59.629612 2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dace71e-73b8-4977-a44f-6f5ef30661ff-credential-socket\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:11:59.629666 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:59.629636 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dace71e-73b8-4977-a44f-6f5ef30661ff-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6dace71e-73b8-4977-a44f-6f5ef30661ff" (UID: "6dace71e-73b8-4977-a44f-6f5ef30661ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:11:59.629913 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:59.629869 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dace71e-73b8-4977-a44f-6f5ef30661ff-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6dace71e-73b8-4977-a44f-6f5ef30661ff" (UID: "6dace71e-73b8-4977-a44f-6f5ef30661ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:11:59.630079 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:59.630046 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dace71e-73b8-4977-a44f-6f5ef30661ff-istio-data" (OuterVolumeSpecName: "istio-data") pod "6dace71e-73b8-4977-a44f-6f5ef30661ff" (UID: "6dace71e-73b8-4977-a44f-6f5ef30661ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:11:59.631505 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:59.631473 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dace71e-73b8-4977-a44f-6f5ef30661ff-istio-token" (OuterVolumeSpecName: "istio-token") pod "6dace71e-73b8-4977-a44f-6f5ef30661ff" (UID: "6dace71e-73b8-4977-a44f-6f5ef30661ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:11:59.631623 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:59.631536 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6dace71e-73b8-4977-a44f-6f5ef30661ff-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6dace71e-73b8-4977-a44f-6f5ef30661ff" (UID: "6dace71e-73b8-4977-a44f-6f5ef30661ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 15:11:59.631671 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:59.631633 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dace71e-73b8-4977-a44f-6f5ef30661ff-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6dace71e-73b8-4977-a44f-6f5ef30661ff" (UID: "6dace71e-73b8-4977-a44f-6f5ef30661ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:11:59.631721 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:59.631688 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dace71e-73b8-4977-a44f-6f5ef30661ff-kube-api-access-j4tbk" (OuterVolumeSpecName: "kube-api-access-j4tbk") pod "6dace71e-73b8-4977-a44f-6f5ef30661ff" (UID: "6dace71e-73b8-4977-a44f-6f5ef30661ff"). InnerVolumeSpecName "kube-api-access-j4tbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:11:59.730665 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:59.730553 2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dace71e-73b8-4977-a44f-6f5ef30661ff-workload-socket\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:11:59.730665 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:59.730602 2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dace71e-73b8-4977-a44f-6f5ef30661ff-istio-envoy\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:11:59.730665 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:59.730613 2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dace71e-73b8-4977-a44f-6f5ef30661ff-istio-data\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:11:59.730665 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:59.730623 2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dace71e-73b8-4977-a44f-6f5ef30661ff-workload-certs\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:11:59.730665 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:59.730632 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4tbk\" (UniqueName: \"kubernetes.io/projected/6dace71e-73b8-4977-a44f-6f5ef30661ff-kube-api-access-j4tbk\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:11:59.730665 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:59.730641 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dace71e-73b8-4977-a44f-6f5ef30661ff-istio-token\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:11:59.730665 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:59.730651 2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dace71e-73b8-4977-a44f-6f5ef30661ff-istio-podinfo\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:11:59.730665 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:59.730658 2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dace71e-73b8-4977-a44f-6f5ef30661ff-istiod-ca-cert\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:11:59.797140 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:59.797103 2555 generic.go:356] "Generic (PLEG): container finished" podID="6dace71e-73b8-4977-a44f-6f5ef30661ff" containerID="f4cdc55f6b9162f1a28ac841e515403fc23b1c1bd641b2fe0772d6dabc9de560" exitCode=0 Jul 07 15:11:59.797317 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:59.797161 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" Jul 07 15:11:59.797317 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:59.797197 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" event={"ID":"6dace71e-73b8-4977-a44f-6f5ef30661ff","Type":"ContainerDied","Data":"f4cdc55f6b9162f1a28ac841e515403fc23b1c1bd641b2fe0772d6dabc9de560"} Jul 07 15:11:59.797317 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:59.797266 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff" event={"ID":"6dace71e-73b8-4977-a44f-6f5ef30661ff","Type":"ContainerDied","Data":"e286114506edbd6a9cc6505ebff0b875f0290708f41d64949a1127d1cdcaf73d"} Jul 07 15:11:59.797317 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:59.797294 2555 scope.go:117] "RemoveContainer" containerID="f4cdc55f6b9162f1a28ac841e515403fc23b1c1bd641b2fe0772d6dabc9de560" Jul 07 15:11:59.807178 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:59.807151 2555 scope.go:117] "RemoveContainer" containerID="6e76985dec8c336cc9b625dd3a71b2fe3935081576d821385736fac12840c047" Jul 07 15:11:59.815776 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:59.815736 2555 scope.go:117] "RemoveContainer" containerID="f4cdc55f6b9162f1a28ac841e515403fc23b1c1bd641b2fe0772d6dabc9de560" Jul 07 15:11:59.816112 ip-10-0-136-19 kubenswrapper[2555]: E0707 15:11:59.816091 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4cdc55f6b9162f1a28ac841e515403fc23b1c1bd641b2fe0772d6dabc9de560\": container with ID starting with f4cdc55f6b9162f1a28ac841e515403fc23b1c1bd641b2fe0772d6dabc9de560 not found: ID does not exist" containerID="f4cdc55f6b9162f1a28ac841e515403fc23b1c1bd641b2fe0772d6dabc9de560" Jul 07 15:11:59.816161 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:59.816122 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4cdc55f6b9162f1a28ac841e515403fc23b1c1bd641b2fe0772d6dabc9de560"} err="failed to get container status \"f4cdc55f6b9162f1a28ac841e515403fc23b1c1bd641b2fe0772d6dabc9de560\": rpc error: code = NotFound desc = could not find container \"f4cdc55f6b9162f1a28ac841e515403fc23b1c1bd641b2fe0772d6dabc9de560\": container with ID starting with f4cdc55f6b9162f1a28ac841e515403fc23b1c1bd641b2fe0772d6dabc9de560 not found: ID does not exist" Jul 07 15:11:59.816161 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:59.816144 2555 scope.go:117] "RemoveContainer" containerID="6e76985dec8c336cc9b625dd3a71b2fe3935081576d821385736fac12840c047" Jul 07 15:11:59.816389 ip-10-0-136-19 kubenswrapper[2555]: E0707 15:11:59.816370 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e76985dec8c336cc9b625dd3a71b2fe3935081576d821385736fac12840c047\": container with ID starting with 6e76985dec8c336cc9b625dd3a71b2fe3935081576d821385736fac12840c047 not found: ID does not exist" containerID="6e76985dec8c336cc9b625dd3a71b2fe3935081576d821385736fac12840c047" Jul 07 15:11:59.816437 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:59.816396 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e76985dec8c336cc9b625dd3a71b2fe3935081576d821385736fac12840c047"} err="failed to get container status \"6e76985dec8c336cc9b625dd3a71b2fe3935081576d821385736fac12840c047\": rpc error: code = NotFound desc = could not find container \"6e76985dec8c336cc9b625dd3a71b2fe3935081576d821385736fac12840c047\": container with ID starting with 6e76985dec8c336cc9b625dd3a71b2fe3935081576d821385736fac12840c047 not found: ID does not exist" Jul 07 15:11:59.821556 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:59.821521 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff"] Jul 07 15:11:59.827175 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:11:59.827139 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-45adb8-openshift-default-56f787878f-45dff"] Jul 07 15:12:01.042539 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:01.042495 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6dace71e-73b8-4977-a44f-6f5ef30661ff" path="/var/lib/kubelet/pods/6dace71e-73b8-4977-a44f-6f5ef30661ff/volumes" Jul 07 15:12:02.387249 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.387210 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn"] Jul 07 15:12:02.387838 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.387816 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44cf7a97-38d9-4f56-b106-d43b3535edc8" containerName="maas-api" Jul 07 15:12:02.387929 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.387840 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="44cf7a97-38d9-4f56-b106-d43b3535edc8" containerName="maas-api" Jul 07 15:12:02.387929 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.387860 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79f774fc-b54f-49e9-ae60-1fdf5ca78c89" containerName="maas-api" Jul 07 15:12:02.387929 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.387868 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="79f774fc-b54f-49e9-ae60-1fdf5ca78c89" containerName="maas-api" Jul 07 15:12:02.387929 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.387878 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dace71e-73b8-4977-a44f-6f5ef30661ff" containerName="istio-proxy" Jul 07 15:12:02.387929 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.387886 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dace71e-73b8-4977-a44f-6f5ef30661ff" containerName="istio-proxy" Jul 07 15:12:02.387929 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.387902 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dace71e-73b8-4977-a44f-6f5ef30661ff" containerName="istio-proxy" Jul 07 15:12:02.387929 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.387911 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dace71e-73b8-4977-a44f-6f5ef30661ff" containerName="istio-proxy" Jul 07 15:12:02.388275 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.388003 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="44cf7a97-38d9-4f56-b106-d43b3535edc8" containerName="maas-api" Jul 07 15:12:02.388275 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.388015 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dace71e-73b8-4977-a44f-6f5ef30661ff" containerName="istio-proxy" Jul 07 15:12:02.388275 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.388026 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dace71e-73b8-4977-a44f-6f5ef30661ff" containerName="istio-proxy" Jul 07 15:12:02.388275 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.388035 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="79f774fc-b54f-49e9-ae60-1fdf5ca78c89" containerName="maas-api" Jul 07 15:12:02.392901 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.392872 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" Jul 07 15:12:02.395858 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.395828 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-dad003-openshift-default-dockercfg-fn4nx\"" Jul 07 15:12:02.408092 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.408058 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn"] Jul 07 15:12:02.565075 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.565035 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da9f0899-8b8a-4e2d-9596-5762553020ee-istio-data\") pod \"e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn\" (UID: \"da9f0899-8b8a-4e2d-9596-5762553020ee\") " pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" Jul 07 15:12:02.565075 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.565078 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da9f0899-8b8a-4e2d-9596-5762553020ee-istio-token\") pod \"e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn\" (UID: \"da9f0899-8b8a-4e2d-9596-5762553020ee\") " pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" Jul 07 15:12:02.565332 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.565118 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk22w\" (UniqueName: \"kubernetes.io/projected/da9f0899-8b8a-4e2d-9596-5762553020ee-kube-api-access-bk22w\") pod \"e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn\" (UID: \"da9f0899-8b8a-4e2d-9596-5762553020ee\") " pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" Jul 07 15:12:02.565332 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.565242 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da9f0899-8b8a-4e2d-9596-5762553020ee-workload-certs\") pod \"e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn\" (UID: \"da9f0899-8b8a-4e2d-9596-5762553020ee\") " pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" Jul 07 15:12:02.565332 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.565273 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9f0899-8b8a-4e2d-9596-5762553020ee-workload-socket\") pod \"e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn\" (UID: \"da9f0899-8b8a-4e2d-9596-5762553020ee\") " pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" Jul 07 15:12:02.565332 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.565291 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da9f0899-8b8a-4e2d-9596-5762553020ee-istio-envoy\") pod \"e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn\" (UID: \"da9f0899-8b8a-4e2d-9596-5762553020ee\") " pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" Jul 07 15:12:02.565516 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.565349 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9f0899-8b8a-4e2d-9596-5762553020ee-credential-socket\") pod \"e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn\" (UID: \"da9f0899-8b8a-4e2d-9596-5762553020ee\") " pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" Jul 07 15:12:02.565516 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.565401 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da9f0899-8b8a-4e2d-9596-5762553020ee-istio-podinfo\") pod \"e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn\" (UID: \"da9f0899-8b8a-4e2d-9596-5762553020ee\") " pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" Jul 07 15:12:02.565516 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.565492 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da9f0899-8b8a-4e2d-9596-5762553020ee-istiod-ca-cert\") pod \"e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn\" (UID: \"da9f0899-8b8a-4e2d-9596-5762553020ee\") " pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" Jul 07 15:12:02.666274 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.666172 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da9f0899-8b8a-4e2d-9596-5762553020ee-workload-certs\") pod \"e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn\" (UID: \"da9f0899-8b8a-4e2d-9596-5762553020ee\") " pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" Jul 07 15:12:02.666274 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.666212 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9f0899-8b8a-4e2d-9596-5762553020ee-workload-socket\") pod \"e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn\" (UID: \"da9f0899-8b8a-4e2d-9596-5762553020ee\") " pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" Jul 07 15:12:02.666274 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.666233 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da9f0899-8b8a-4e2d-9596-5762553020ee-istio-envoy\") pod \"e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn\" (UID: \"da9f0899-8b8a-4e2d-9596-5762553020ee\") " pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" Jul 07 15:12:02.666577 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.666354 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9f0899-8b8a-4e2d-9596-5762553020ee-credential-socket\") pod \"e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn\" (UID: \"da9f0899-8b8a-4e2d-9596-5762553020ee\") " pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" Jul 07 15:12:02.666577 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.666409 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da9f0899-8b8a-4e2d-9596-5762553020ee-istio-podinfo\") pod \"e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn\" (UID: \"da9f0899-8b8a-4e2d-9596-5762553020ee\") " pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" Jul 07 15:12:02.666577 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.666493 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da9f0899-8b8a-4e2d-9596-5762553020ee-istiod-ca-cert\") pod \"e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn\" (UID: \"da9f0899-8b8a-4e2d-9596-5762553020ee\") " pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" Jul 07 15:12:02.666577 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.666531 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da9f0899-8b8a-4e2d-9596-5762553020ee-istio-data\") pod \"e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn\" (UID: \"da9f0899-8b8a-4e2d-9596-5762553020ee\") " pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" Jul 07 15:12:02.666577 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.666561 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da9f0899-8b8a-4e2d-9596-5762553020ee-istio-token\") pod \"e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn\" (UID: \"da9f0899-8b8a-4e2d-9596-5762553020ee\") " pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" Jul 07 15:12:02.666863 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.666603 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk22w\" (UniqueName: \"kubernetes.io/projected/da9f0899-8b8a-4e2d-9596-5762553020ee-kube-api-access-bk22w\") pod \"e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn\" (UID: \"da9f0899-8b8a-4e2d-9596-5762553020ee\") " pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" Jul 07 15:12:02.666863 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.666661 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da9f0899-8b8a-4e2d-9596-5762553020ee-workload-certs\") pod \"e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn\" (UID: \"da9f0899-8b8a-4e2d-9596-5762553020ee\") " pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" Jul 07 15:12:02.666863 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.666799 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9f0899-8b8a-4e2d-9596-5762553020ee-credential-socket\") pod \"e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn\" (UID: \"da9f0899-8b8a-4e2d-9596-5762553020ee\") " pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" Jul 07 15:12:02.666863 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.666805 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9f0899-8b8a-4e2d-9596-5762553020ee-workload-socket\") pod \"e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn\" (UID: \"da9f0899-8b8a-4e2d-9596-5762553020ee\") " pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" Jul 07 15:12:02.667047 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.666873 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da9f0899-8b8a-4e2d-9596-5762553020ee-istio-data\") pod \"e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn\" (UID: \"da9f0899-8b8a-4e2d-9596-5762553020ee\") " pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" Jul 07 15:12:02.667392 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.667373 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da9f0899-8b8a-4e2d-9596-5762553020ee-istiod-ca-cert\") pod \"e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn\" (UID: \"da9f0899-8b8a-4e2d-9596-5762553020ee\") " pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" Jul 07 15:12:02.668569 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.668547 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da9f0899-8b8a-4e2d-9596-5762553020ee-istio-envoy\") pod \"e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn\" (UID: \"da9f0899-8b8a-4e2d-9596-5762553020ee\") " pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" Jul 07 15:12:02.669012 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.668988 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da9f0899-8b8a-4e2d-9596-5762553020ee-istio-podinfo\") pod \"e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn\" (UID: \"da9f0899-8b8a-4e2d-9596-5762553020ee\") " pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" Jul 07 15:12:02.675474 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.675449 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da9f0899-8b8a-4e2d-9596-5762553020ee-istio-token\") pod \"e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn\" (UID: \"da9f0899-8b8a-4e2d-9596-5762553020ee\") " pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" Jul 07 15:12:02.675778 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.675734 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk22w\" (UniqueName: \"kubernetes.io/projected/da9f0899-8b8a-4e2d-9596-5762553020ee-kube-api-access-bk22w\") pod \"e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn\" (UID: \"da9f0899-8b8a-4e2d-9596-5762553020ee\") " pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" Jul 07 15:12:02.703674 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.703629 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" Jul 07 15:12:02.849340 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.849314 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn"] Jul 07 15:12:02.851653 ip-10-0-136-19 kubenswrapper[2555]: W0707 15:12:02.851629 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda9f0899_8b8a_4e2d_9596_5762553020ee.slice/crio-a3122c4c19cea1d72210bbcefb3cdc4129b66531127129efdff7558fa5225691 WatchSource:0}: Error finding container a3122c4c19cea1d72210bbcefb3cdc4129b66531127129efdff7558fa5225691: Status 404 returned error can't find the container with id a3122c4c19cea1d72210bbcefb3cdc4129b66531127129efdff7558fa5225691 Jul 07 15:12:02.854307 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.854279 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:12:02.854372 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.854345 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:12:02.854410 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:02.854373 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:12:03.820330 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:03.820293 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" event={"ID":"da9f0899-8b8a-4e2d-9596-5762553020ee","Type":"ContainerStarted","Data":"02eb27d0334b58cb61e399619caf4aae21dece01c8e7dd58fa6a199f145a5609"} Jul 07 15:12:03.820829 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:03.820350 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" event={"ID":"da9f0899-8b8a-4e2d-9596-5762553020ee","Type":"ContainerStarted","Data":"a3122c4c19cea1d72210bbcefb3cdc4129b66531127129efdff7558fa5225691"} Jul 07 15:12:03.849444 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:03.849393 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" podStartSLOduration=1.849373345 podStartE2EDuration="1.849373345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:12:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:12:03.84821934 +0000 UTC m=+2265.272405829" watchObservedRunningTime="2026-07-07 15:12:03.849373345 +0000 UTC m=+2265.273559820" Jul 07 15:12:04.703936 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:04.703894 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" Jul 07 15:12:04.709038 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:04.709006 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" Jul 07 15:12:04.826261 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:04.826225 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" Jul 07 15:12:04.827314 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:04.827291 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" Jul 07 15:12:05.890015 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:05.889969 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-a-dad003-869564fdcb-5984r"] Jul 07 15:12:05.893708 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:05.893684 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-dad003-869564fdcb-5984r" Jul 07 15:12:05.896653 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:05.896627 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-a-dad003-serving-cert\"" Jul 07 15:12:05.912790 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:05.912741 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-dad003-869564fdcb-5984r"] Jul 07 15:12:05.999200 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:05.999157 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prjhk\" (UniqueName: \"kubernetes.io/projected/9f1c3d81-9c34-4845-a386-f3bac463b191-kube-api-access-prjhk\") pod \"maas-api-e2e-api-a-dad003-869564fdcb-5984r\" (UID: \"9f1c3d81-9c34-4845-a386-f3bac463b191\") " pod="opendatahub/maas-api-e2e-api-a-dad003-869564fdcb-5984r" Jul 07 15:12:05.999401 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:05.999215 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f1c3d81-9c34-4845-a386-f3bac463b191-openshift-service-ca\") pod \"maas-api-e2e-api-a-dad003-869564fdcb-5984r\" (UID: \"9f1c3d81-9c34-4845-a386-f3bac463b191\") " pod="opendatahub/maas-api-e2e-api-a-dad003-869564fdcb-5984r" Jul 07 15:12:05.999401 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:05.999310 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f1c3d81-9c34-4845-a386-f3bac463b191-maas-api-tls\") pod \"maas-api-e2e-api-a-dad003-869564fdcb-5984r\" (UID: \"9f1c3d81-9c34-4845-a386-f3bac463b191\") " pod="opendatahub/maas-api-e2e-api-a-dad003-869564fdcb-5984r" Jul 07 15:12:06.100347 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:06.100301 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prjhk\" (UniqueName: \"kubernetes.io/projected/9f1c3d81-9c34-4845-a386-f3bac463b191-kube-api-access-prjhk\") pod \"maas-api-e2e-api-a-dad003-869564fdcb-5984r\" (UID: \"9f1c3d81-9c34-4845-a386-f3bac463b191\") " pod="opendatahub/maas-api-e2e-api-a-dad003-869564fdcb-5984r" Jul 07 15:12:06.100536 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:06.100357 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f1c3d81-9c34-4845-a386-f3bac463b191-openshift-service-ca\") pod \"maas-api-e2e-api-a-dad003-869564fdcb-5984r\" (UID: \"9f1c3d81-9c34-4845-a386-f3bac463b191\") " pod="opendatahub/maas-api-e2e-api-a-dad003-869564fdcb-5984r" Jul 07 15:12:06.100536 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:06.100386 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f1c3d81-9c34-4845-a386-f3bac463b191-maas-api-tls\") pod \"maas-api-e2e-api-a-dad003-869564fdcb-5984r\" (UID: \"9f1c3d81-9c34-4845-a386-f3bac463b191\") " pod="opendatahub/maas-api-e2e-api-a-dad003-869564fdcb-5984r" Jul 07 15:12:06.100536 ip-10-0-136-19 kubenswrapper[2555]: E0707 15:12:06.100526 2555 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-a-dad003-serving-cert: secret "maas-api-e2e-api-a-dad003-serving-cert" not found Jul 07 15:12:06.100643 ip-10-0-136-19 kubenswrapper[2555]: E0707 15:12:06.100580 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f1c3d81-9c34-4845-a386-f3bac463b191-maas-api-tls podName:9f1c3d81-9c34-4845-a386-f3bac463b191 nodeName:}" failed. No retries permitted until 2026-07-07 15:12:06.600564544 +0000 UTC m=+2268.024750999 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9f1c3d81-9c34-4845-a386-f3bac463b191-maas-api-tls") pod "maas-api-e2e-api-a-dad003-869564fdcb-5984r" (UID: "9f1c3d81-9c34-4845-a386-f3bac463b191") : secret "maas-api-e2e-api-a-dad003-serving-cert" not found Jul 07 15:12:06.101099 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:06.101077 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f1c3d81-9c34-4845-a386-f3bac463b191-openshift-service-ca\") pod \"maas-api-e2e-api-a-dad003-869564fdcb-5984r\" (UID: \"9f1c3d81-9c34-4845-a386-f3bac463b191\") " pod="opendatahub/maas-api-e2e-api-a-dad003-869564fdcb-5984r" Jul 07 15:12:06.115733 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:06.115701 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prjhk\" (UniqueName: \"kubernetes.io/projected/9f1c3d81-9c34-4845-a386-f3bac463b191-kube-api-access-prjhk\") pod \"maas-api-e2e-api-a-dad003-869564fdcb-5984r\" (UID: \"9f1c3d81-9c34-4845-a386-f3bac463b191\") " pod="opendatahub/maas-api-e2e-api-a-dad003-869564fdcb-5984r" Jul 07 15:12:06.606468 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:06.606424 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f1c3d81-9c34-4845-a386-f3bac463b191-maas-api-tls\") pod \"maas-api-e2e-api-a-dad003-869564fdcb-5984r\" (UID: \"9f1c3d81-9c34-4845-a386-f3bac463b191\") " pod="opendatahub/maas-api-e2e-api-a-dad003-869564fdcb-5984r" Jul 07 15:12:06.608967 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:06.608940 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f1c3d81-9c34-4845-a386-f3bac463b191-maas-api-tls\") pod \"maas-api-e2e-api-a-dad003-869564fdcb-5984r\" (UID: \"9f1c3d81-9c34-4845-a386-f3bac463b191\") " pod="opendatahub/maas-api-e2e-api-a-dad003-869564fdcb-5984r" Jul 07 15:12:06.805101 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:06.805055 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-dad003-869564fdcb-5984r" Jul 07 15:12:06.958307 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:06.958278 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-dad003-869564fdcb-5984r"] Jul 07 15:12:06.959938 ip-10-0-136-19 kubenswrapper[2555]: W0707 15:12:06.959912 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f1c3d81_9c34_4845_a386_f3bac463b191.slice/crio-b9348ebe2bc3900b6e695222cf3dff4d270d6c8920729450f958e47bbe31fd51 WatchSource:0}: Error finding container b9348ebe2bc3900b6e695222cf3dff4d270d6c8920729450f958e47bbe31fd51: Status 404 returned error can't find the container with id b9348ebe2bc3900b6e695222cf3dff4d270d6c8920729450f958e47bbe31fd51 Jul 07 15:12:07.844138 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:07.844101 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-dad003-869564fdcb-5984r" event={"ID":"9f1c3d81-9c34-4845-a386-f3bac463b191","Type":"ContainerStarted","Data":"72852594179db58c94d0cc967e6aba9214fe16d17396ded8d715fc396a784fec"} Jul 07 15:12:07.844336 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:07.844159 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-dad003-869564fdcb-5984r" event={"ID":"9f1c3d81-9c34-4845-a386-f3bac463b191","Type":"ContainerStarted","Data":"b9348ebe2bc3900b6e695222cf3dff4d270d6c8920729450f958e47bbe31fd51"} Jul 07 15:12:07.844336 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:07.844196 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-a-dad003-869564fdcb-5984r" Jul 07 15:12:07.882655 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:07.882596 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-a-dad003-869564fdcb-5984r" podStartSLOduration=2.34177327 podStartE2EDuration="2.882578971s" totalImagesPullingTime="540.805701ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:12:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:12:07.87968506 +0000 UTC m=+2269.303871532" watchObservedRunningTime="2026-07-07 15:12:07.882578971 +0000 UTC m=+2269.306765480" Jul 07 15:12:11.061926 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:11.061879 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45"] Jul 07 15:12:11.076889 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:11.076851 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" Jul 07 15:12:11.080029 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:11.079994 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45"] Jul 07 15:12:11.080232 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:11.080213 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-f21b85-openshift-default-dockercfg-jhk9w\"" Jul 07 15:12:11.254296 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:11.254248 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/009e99b4-b67a-472e-b8a2-4536d9d59bc1-istiod-ca-cert\") pod \"e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45\" (UID: \"009e99b4-b67a-472e-b8a2-4536d9d59bc1\") " pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" Jul 07 15:12:11.254645 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:11.254623 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/009e99b4-b67a-472e-b8a2-4536d9d59bc1-istio-envoy\") pod \"e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45\" (UID: \"009e99b4-b67a-472e-b8a2-4536d9d59bc1\") " pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" Jul 07 15:12:11.254829 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:11.254812 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/009e99b4-b67a-472e-b8a2-4536d9d59bc1-istio-token\") pod \"e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45\" (UID: \"009e99b4-b67a-472e-b8a2-4536d9d59bc1\") " pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" Jul 07 15:12:11.254984 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:11.254968 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nxgd\" (UniqueName: \"kubernetes.io/projected/009e99b4-b67a-472e-b8a2-4536d9d59bc1-kube-api-access-6nxgd\") pod \"e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45\" (UID: \"009e99b4-b67a-472e-b8a2-4536d9d59bc1\") " pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" Jul 07 15:12:11.255129 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:11.255113 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/009e99b4-b67a-472e-b8a2-4536d9d59bc1-istio-podinfo\") pod \"e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45\" (UID: \"009e99b4-b67a-472e-b8a2-4536d9d59bc1\") " pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" Jul 07 15:12:11.255286 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:11.255271 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/009e99b4-b67a-472e-b8a2-4536d9d59bc1-credential-socket\") pod \"e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45\" (UID: \"009e99b4-b67a-472e-b8a2-4536d9d59bc1\") " pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" Jul 07 15:12:11.255433 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:11.255419 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/009e99b4-b67a-472e-b8a2-4536d9d59bc1-workload-socket\") pod \"e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45\" (UID: \"009e99b4-b67a-472e-b8a2-4536d9d59bc1\") " pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" Jul 07 15:12:11.255627 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:11.255612 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/009e99b4-b67a-472e-b8a2-4536d9d59bc1-workload-certs\") pod \"e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45\" (UID: \"009e99b4-b67a-472e-b8a2-4536d9d59bc1\") " pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" Jul 07 15:12:11.255809 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:11.255794 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/009e99b4-b67a-472e-b8a2-4536d9d59bc1-istio-data\") pod \"e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45\" (UID: \"009e99b4-b67a-472e-b8a2-4536d9d59bc1\") " pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" Jul 07 15:12:11.356673 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:11.356627 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/009e99b4-b67a-472e-b8a2-4536d9d59bc1-workload-socket\") pod \"e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45\" (UID: \"009e99b4-b67a-472e-b8a2-4536d9d59bc1\") " pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" Jul 07 15:12:11.356927 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:11.356737 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/009e99b4-b67a-472e-b8a2-4536d9d59bc1-workload-certs\") pod \"e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45\" (UID: \"009e99b4-b67a-472e-b8a2-4536d9d59bc1\") " pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" Jul 07 15:12:11.356927 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:11.356811 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/009e99b4-b67a-472e-b8a2-4536d9d59bc1-istio-data\") pod \"e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45\" (UID: \"009e99b4-b67a-472e-b8a2-4536d9d59bc1\") " pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" Jul 07 15:12:11.356927 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:11.356850 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/009e99b4-b67a-472e-b8a2-4536d9d59bc1-istiod-ca-cert\") pod \"e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45\" (UID: \"009e99b4-b67a-472e-b8a2-4536d9d59bc1\") " pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" Jul 07 15:12:11.356927 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:11.356879 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/009e99b4-b67a-472e-b8a2-4536d9d59bc1-istio-envoy\") pod \"e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45\" (UID: \"009e99b4-b67a-472e-b8a2-4536d9d59bc1\") " pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" Jul 07 15:12:11.356927 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:11.356907 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/009e99b4-b67a-472e-b8a2-4536d9d59bc1-istio-token\") pod \"e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45\" (UID: \"009e99b4-b67a-472e-b8a2-4536d9d59bc1\") " pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" Jul 07 15:12:11.357223 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:11.356942 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nxgd\" (UniqueName: \"kubernetes.io/projected/009e99b4-b67a-472e-b8a2-4536d9d59bc1-kube-api-access-6nxgd\") pod \"e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45\" (UID: \"009e99b4-b67a-472e-b8a2-4536d9d59bc1\") " pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" Jul 07 15:12:11.357223 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:11.356975 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/009e99b4-b67a-472e-b8a2-4536d9d59bc1-istio-podinfo\") pod \"e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45\" (UID: \"009e99b4-b67a-472e-b8a2-4536d9d59bc1\") " pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" Jul 07 15:12:11.357223 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:11.357015 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/009e99b4-b67a-472e-b8a2-4536d9d59bc1-credential-socket\") pod \"e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45\" (UID: \"009e99b4-b67a-472e-b8a2-4536d9d59bc1\") " pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" Jul 07 15:12:11.357223 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:11.357148 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/009e99b4-b67a-472e-b8a2-4536d9d59bc1-workload-socket\") pod \"e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45\" (UID: \"009e99b4-b67a-472e-b8a2-4536d9d59bc1\") " pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" Jul 07 15:12:11.357439 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:11.357367 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/009e99b4-b67a-472e-b8a2-4536d9d59bc1-credential-socket\") pod \"e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45\" (UID: \"009e99b4-b67a-472e-b8a2-4536d9d59bc1\") " pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" Jul 07 15:12:11.358448 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:11.357790 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/009e99b4-b67a-472e-b8a2-4536d9d59bc1-istio-data\") pod \"e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45\" (UID: \"009e99b4-b67a-472e-b8a2-4536d9d59bc1\") " pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" Jul 07 15:12:11.358448 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:11.358086 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/009e99b4-b67a-472e-b8a2-4536d9d59bc1-workload-certs\") pod \"e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45\" (UID: \"009e99b4-b67a-472e-b8a2-4536d9d59bc1\") " pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" Jul 07 15:12:11.358448 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:11.358394 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/009e99b4-b67a-472e-b8a2-4536d9d59bc1-istiod-ca-cert\") pod \"e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45\" (UID: \"009e99b4-b67a-472e-b8a2-4536d9d59bc1\") " pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" Jul 07 15:12:11.360049 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:11.360023 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/009e99b4-b67a-472e-b8a2-4536d9d59bc1-istio-envoy\") pod \"e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45\" (UID: \"009e99b4-b67a-472e-b8a2-4536d9d59bc1\") " pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" Jul 07 15:12:11.362505 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:11.362477 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/009e99b4-b67a-472e-b8a2-4536d9d59bc1-istio-podinfo\") pod \"e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45\" (UID: \"009e99b4-b67a-472e-b8a2-4536d9d59bc1\") " pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" Jul 07 15:12:11.365362 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:11.365334 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/009e99b4-b67a-472e-b8a2-4536d9d59bc1-istio-token\") pod \"e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45\" (UID: \"009e99b4-b67a-472e-b8a2-4536d9d59bc1\") " pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" Jul 07 15:12:11.365493 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:11.365474 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nxgd\" (UniqueName: \"kubernetes.io/projected/009e99b4-b67a-472e-b8a2-4536d9d59bc1-kube-api-access-6nxgd\") pod \"e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45\" (UID: \"009e99b4-b67a-472e-b8a2-4536d9d59bc1\") " pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" Jul 07 15:12:11.388438 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:11.388382 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" Jul 07 15:12:11.585659 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:11.585632 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45"] Jul 07 15:12:11.587297 ip-10-0-136-19 kubenswrapper[2555]: W0707 15:12:11.587273 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod009e99b4_b67a_472e_b8a2_4536d9d59bc1.slice/crio-863243bf219b2c6a31bf128e1d9cc258c025bba15973cfe209b40915317f9706 WatchSource:0}: Error finding container 863243bf219b2c6a31bf128e1d9cc258c025bba15973cfe209b40915317f9706: Status 404 returned error can't find the container with id 863243bf219b2c6a31bf128e1d9cc258c025bba15973cfe209b40915317f9706 Jul 07 15:12:11.589563 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:11.589528 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:12:11.589669 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:11.589612 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:12:11.589669 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:11.589657 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:12:11.871976 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:11.871866 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" event={"ID":"009e99b4-b67a-472e-b8a2-4536d9d59bc1","Type":"ContainerStarted","Data":"40410464836dfb0f1c4b10a83027efb8fddf6bc8475fc6ff0830af98a4607627"} Jul 07 15:12:11.871976 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:11.871933 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" event={"ID":"009e99b4-b67a-472e-b8a2-4536d9d59bc1","Type":"ContainerStarted","Data":"863243bf219b2c6a31bf128e1d9cc258c025bba15973cfe209b40915317f9706"} Jul 07 15:12:11.896326 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:11.896259 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" podStartSLOduration=0.89623598 podStartE2EDuration="896.23598ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:12:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:12:11.895569828 +0000 UTC m=+2273.319756309" watchObservedRunningTime="2026-07-07 15:12:11.89623598 +0000 UTC m=+2273.320422455" Jul 07 15:12:12.389447 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:12.389408 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" Jul 07 15:12:12.394887 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:12.394855 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" Jul 07 15:12:12.877082 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:12.877047 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" Jul 07 15:12:12.878150 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:12.878133 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" Jul 07 15:12:13.854973 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:13.854945 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-a-dad003-869564fdcb-5984r" Jul 07 15:12:13.918771 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:13.918720 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-b-f21b85-65786b477f-tq9qc"] Jul 07 15:12:13.922625 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:13.922596 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-f21b85-65786b477f-tq9qc" Jul 07 15:12:13.926170 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:13.926121 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-b-f21b85-serving-cert\"" Jul 07 15:12:13.937579 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:13.937536 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-f21b85-65786b477f-tq9qc"] Jul 07 15:12:13.983423 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:13.983371 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9b2da02-3359-41a4-8a05-2ff040be5b5e-maas-api-tls\") pod \"maas-api-e2e-api-b-f21b85-65786b477f-tq9qc\" (UID: \"e9b2da02-3359-41a4-8a05-2ff040be5b5e\") " pod="opendatahub/maas-api-e2e-api-b-f21b85-65786b477f-tq9qc" Jul 07 15:12:13.983423 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:13.983424 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9b2da02-3359-41a4-8a05-2ff040be5b5e-openshift-service-ca\") pod \"maas-api-e2e-api-b-f21b85-65786b477f-tq9qc\" (UID: \"e9b2da02-3359-41a4-8a05-2ff040be5b5e\") " pod="opendatahub/maas-api-e2e-api-b-f21b85-65786b477f-tq9qc" Jul 07 15:12:13.983678 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:13.983501 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjbw8\" (UniqueName: \"kubernetes.io/projected/e9b2da02-3359-41a4-8a05-2ff040be5b5e-kube-api-access-mjbw8\") pod \"maas-api-e2e-api-b-f21b85-65786b477f-tq9qc\" (UID: \"e9b2da02-3359-41a4-8a05-2ff040be5b5e\") " pod="opendatahub/maas-api-e2e-api-b-f21b85-65786b477f-tq9qc" Jul 07 15:12:14.084342 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:14.084288 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9b2da02-3359-41a4-8a05-2ff040be5b5e-maas-api-tls\") pod \"maas-api-e2e-api-b-f21b85-65786b477f-tq9qc\" (UID: \"e9b2da02-3359-41a4-8a05-2ff040be5b5e\") " pod="opendatahub/maas-api-e2e-api-b-f21b85-65786b477f-tq9qc" Jul 07 15:12:14.084342 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:14.084345 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9b2da02-3359-41a4-8a05-2ff040be5b5e-openshift-service-ca\") pod \"maas-api-e2e-api-b-f21b85-65786b477f-tq9qc\" (UID: \"e9b2da02-3359-41a4-8a05-2ff040be5b5e\") " pod="opendatahub/maas-api-e2e-api-b-f21b85-65786b477f-tq9qc" Jul 07 15:12:14.084632 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:14.084424 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjbw8\" (UniqueName: \"kubernetes.io/projected/e9b2da02-3359-41a4-8a05-2ff040be5b5e-kube-api-access-mjbw8\") pod \"maas-api-e2e-api-b-f21b85-65786b477f-tq9qc\" (UID: \"e9b2da02-3359-41a4-8a05-2ff040be5b5e\") " pod="opendatahub/maas-api-e2e-api-b-f21b85-65786b477f-tq9qc" Jul 07 15:12:14.084632 ip-10-0-136-19 kubenswrapper[2555]: E0707 15:12:14.084466 2555 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-b-f21b85-serving-cert: secret "maas-api-e2e-api-b-f21b85-serving-cert" not found Jul 07 15:12:14.084632 ip-10-0-136-19 kubenswrapper[2555]: E0707 15:12:14.084546 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9b2da02-3359-41a4-8a05-2ff040be5b5e-maas-api-tls podName:e9b2da02-3359-41a4-8a05-2ff040be5b5e nodeName:}" failed. No retries permitted until 2026-07-07 15:12:14.584523709 +0000 UTC m=+2276.008710183 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e9b2da02-3359-41a4-8a05-2ff040be5b5e-maas-api-tls") pod "maas-api-e2e-api-b-f21b85-65786b477f-tq9qc" (UID: "e9b2da02-3359-41a4-8a05-2ff040be5b5e") : secret "maas-api-e2e-api-b-f21b85-serving-cert" not found Jul 07 15:12:14.085411 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:14.085388 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9b2da02-3359-41a4-8a05-2ff040be5b5e-openshift-service-ca\") pod \"maas-api-e2e-api-b-f21b85-65786b477f-tq9qc\" (UID: \"e9b2da02-3359-41a4-8a05-2ff040be5b5e\") " pod="opendatahub/maas-api-e2e-api-b-f21b85-65786b477f-tq9qc" Jul 07 15:12:14.095999 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:14.095968 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjbw8\" (UniqueName: \"kubernetes.io/projected/e9b2da02-3359-41a4-8a05-2ff040be5b5e-kube-api-access-mjbw8\") pod \"maas-api-e2e-api-b-f21b85-65786b477f-tq9qc\" (UID: \"e9b2da02-3359-41a4-8a05-2ff040be5b5e\") " pod="opendatahub/maas-api-e2e-api-b-f21b85-65786b477f-tq9qc" Jul 07 15:12:14.590200 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:14.590147 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9b2da02-3359-41a4-8a05-2ff040be5b5e-maas-api-tls\") pod \"maas-api-e2e-api-b-f21b85-65786b477f-tq9qc\" (UID: \"e9b2da02-3359-41a4-8a05-2ff040be5b5e\") " pod="opendatahub/maas-api-e2e-api-b-f21b85-65786b477f-tq9qc" Jul 07 15:12:14.592688 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:14.592663 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9b2da02-3359-41a4-8a05-2ff040be5b5e-maas-api-tls\") pod \"maas-api-e2e-api-b-f21b85-65786b477f-tq9qc\" (UID: \"e9b2da02-3359-41a4-8a05-2ff040be5b5e\") " pod="opendatahub/maas-api-e2e-api-b-f21b85-65786b477f-tq9qc" Jul 07 15:12:14.837891 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:14.837854 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-f21b85-65786b477f-tq9qc" Jul 07 15:12:14.997514 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:14.997488 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-f21b85-65786b477f-tq9qc"] Jul 07 15:12:14.999631 ip-10-0-136-19 kubenswrapper[2555]: W0707 15:12:14.999601 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9b2da02_3359_41a4_8a05_2ff040be5b5e.slice/crio-7102f42227883dc8a0392d93e98820ebfbd6ddd71cc428c6efeba2d4990a238e WatchSource:0}: Error finding container 7102f42227883dc8a0392d93e98820ebfbd6ddd71cc428c6efeba2d4990a238e: Status 404 returned error can't find the container with id 7102f42227883dc8a0392d93e98820ebfbd6ddd71cc428c6efeba2d4990a238e Jul 07 15:12:15.896197 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:15.896160 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-f21b85-65786b477f-tq9qc" event={"ID":"e9b2da02-3359-41a4-8a05-2ff040be5b5e","Type":"ContainerStarted","Data":"50758f9096939f4810bfc130d0e2e4cd8fd048b3f75eea62618c8713c62c1054"} Jul 07 15:12:15.896390 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:15.896255 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-f21b85-65786b477f-tq9qc" event={"ID":"e9b2da02-3359-41a4-8a05-2ff040be5b5e","Type":"ContainerStarted","Data":"7102f42227883dc8a0392d93e98820ebfbd6ddd71cc428c6efeba2d4990a238e"} Jul 07 15:12:15.896390 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:15.896307 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-b-f21b85-65786b477f-tq9qc" Jul 07 15:12:15.917643 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:15.917581 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-b-f21b85-65786b477f-tq9qc" podStartSLOduration=2.4164106309999998 podStartE2EDuration="2.917562504s" totalImagesPullingTime="501.151873ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:12:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:12:15.915481915 +0000 UTC m=+2277.339668391" watchObservedRunningTime="2026-07-07 15:12:15.917562504 +0000 UTC m=+2277.341748979" Jul 07 15:12:16.903451 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:16.903413 2555 generic.go:356] "Generic (PLEG): container finished" podID="da9f0899-8b8a-4e2d-9596-5762553020ee" containerID="02eb27d0334b58cb61e399619caf4aae21dece01c8e7dd58fa6a199f145a5609" exitCode=0 Jul 07 15:12:16.904023 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:16.903483 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" event={"ID":"da9f0899-8b8a-4e2d-9596-5762553020ee","Type":"ContainerDied","Data":"02eb27d0334b58cb61e399619caf4aae21dece01c8e7dd58fa6a199f145a5609"} Jul 07 15:12:16.904023 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:16.903958 2555 scope.go:117] "RemoveContainer" containerID="02eb27d0334b58cb61e399619caf4aae21dece01c8e7dd58fa6a199f145a5609" Jul 07 15:12:16.904959 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:16.904933 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:12:16.905034 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:16.904998 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:12:16.905034 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:16.905031 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:12:17.915153 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:17.915117 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" event={"ID":"da9f0899-8b8a-4e2d-9596-5762553020ee","Type":"ContainerStarted","Data":"f650896ecbcd58b7a4ee5ff431dbd8eb2a89841c21cad1664a543dcc2e687c5f"} Jul 07 15:12:18.703824 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:18.703789 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" Jul 07 15:12:18.705062 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:18.705035 2555 patch_prober.go:28] interesting pod/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 07 15:12:18.705188 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:18.705098 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" podUID="da9f0899-8b8a-4e2d-9596-5762553020ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 07 15:12:19.704649 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:19.704610 2555 patch_prober.go:28] interesting pod/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 07 15:12:19.705096 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:19.704675 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" podUID="da9f0899-8b8a-4e2d-9596-5762553020ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 07 15:12:19.826432 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:19.826391 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" Jul 07 15:12:20.704235 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:20.704189 2555 patch_prober.go:28] interesting pod/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 07 15:12:20.704433 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:20.704266 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" podUID="da9f0899-8b8a-4e2d-9596-5762553020ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 07 15:12:21.704364 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:21.704319 2555 patch_prober.go:28] interesting pod/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 07 15:12:21.704826 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:21.704384 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" podUID="da9f0899-8b8a-4e2d-9596-5762553020ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 07 15:12:21.909991 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:21.909959 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-b-f21b85-65786b477f-tq9qc" Jul 07 15:12:22.711233 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:22.711195 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" Jul 07 15:12:22.712489 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:22.712466 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" Jul 07 15:12:27.435248 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:27.435207 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-dad003-869564fdcb-5984r"] Jul 07 15:12:27.435808 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:27.435511 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-a-dad003-869564fdcb-5984r" podUID="9f1c3d81-9c34-4845-a386-f3bac463b191" containerName="maas-api" containerID="cri-o://72852594179db58c94d0cc967e6aba9214fe16d17396ded8d715fc396a784fec" gracePeriod=30 Jul 07 15:12:27.707140 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:27.707114 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-dad003-869564fdcb-5984r" Jul 07 15:12:27.828175 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:27.828133 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prjhk\" (UniqueName: \"kubernetes.io/projected/9f1c3d81-9c34-4845-a386-f3bac463b191-kube-api-access-prjhk\") pod \"9f1c3d81-9c34-4845-a386-f3bac463b191\" (UID: \"9f1c3d81-9c34-4845-a386-f3bac463b191\") " Jul 07 15:12:27.828175 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:27.828179 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f1c3d81-9c34-4845-a386-f3bac463b191-maas-api-tls\") pod \"9f1c3d81-9c34-4845-a386-f3bac463b191\" (UID: \"9f1c3d81-9c34-4845-a386-f3bac463b191\") " Jul 07 15:12:27.828436 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:27.828208 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f1c3d81-9c34-4845-a386-f3bac463b191-openshift-service-ca\") pod \"9f1c3d81-9c34-4845-a386-f3bac463b191\" (UID: \"9f1c3d81-9c34-4845-a386-f3bac463b191\") " Jul 07 15:12:27.828670 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:27.828637 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f1c3d81-9c34-4845-a386-f3bac463b191-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9f1c3d81-9c34-4845-a386-f3bac463b191" (UID: "9f1c3d81-9c34-4845-a386-f3bac463b191"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:12:27.830422 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:27.830394 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f1c3d81-9c34-4845-a386-f3bac463b191-kube-api-access-prjhk" (OuterVolumeSpecName: "kube-api-access-prjhk") pod "9f1c3d81-9c34-4845-a386-f3bac463b191" (UID: "9f1c3d81-9c34-4845-a386-f3bac463b191"). InnerVolumeSpecName "kube-api-access-prjhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:12:27.830534 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:27.830483 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f1c3d81-9c34-4845-a386-f3bac463b191-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9f1c3d81-9c34-4845-a386-f3bac463b191" (UID: "9f1c3d81-9c34-4845-a386-f3bac463b191"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:12:27.929798 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:27.929738 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prjhk\" (UniqueName: \"kubernetes.io/projected/9f1c3d81-9c34-4845-a386-f3bac463b191-kube-api-access-prjhk\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:12:27.929798 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:27.929790 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f1c3d81-9c34-4845-a386-f3bac463b191-maas-api-tls\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:12:27.929798 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:27.929799 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f1c3d81-9c34-4845-a386-f3bac463b191-openshift-service-ca\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:12:27.981778 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:27.981658 2555 generic.go:356] "Generic (PLEG): container finished" podID="9f1c3d81-9c34-4845-a386-f3bac463b191" containerID="72852594179db58c94d0cc967e6aba9214fe16d17396ded8d715fc396a784fec" exitCode=0 Jul 07 15:12:27.981778 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:27.981705 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-dad003-869564fdcb-5984r" Jul 07 15:12:27.982040 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:27.981746 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-dad003-869564fdcb-5984r" event={"ID":"9f1c3d81-9c34-4845-a386-f3bac463b191","Type":"ContainerDied","Data":"72852594179db58c94d0cc967e6aba9214fe16d17396ded8d715fc396a784fec"} Jul 07 15:12:27.982040 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:27.981847 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-dad003-869564fdcb-5984r" event={"ID":"9f1c3d81-9c34-4845-a386-f3bac463b191","Type":"ContainerDied","Data":"b9348ebe2bc3900b6e695222cf3dff4d270d6c8920729450f958e47bbe31fd51"} Jul 07 15:12:27.982040 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:27.981877 2555 scope.go:117] "RemoveContainer" containerID="72852594179db58c94d0cc967e6aba9214fe16d17396ded8d715fc396a784fec" Jul 07 15:12:27.992270 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:27.992236 2555 scope.go:117] "RemoveContainer" containerID="72852594179db58c94d0cc967e6aba9214fe16d17396ded8d715fc396a784fec" Jul 07 15:12:27.992646 ip-10-0-136-19 kubenswrapper[2555]: E0707 15:12:27.992622 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72852594179db58c94d0cc967e6aba9214fe16d17396ded8d715fc396a784fec\": container with ID starting with 72852594179db58c94d0cc967e6aba9214fe16d17396ded8d715fc396a784fec not found: ID does not exist" containerID="72852594179db58c94d0cc967e6aba9214fe16d17396ded8d715fc396a784fec" Jul 07 15:12:27.992719 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:27.992659 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72852594179db58c94d0cc967e6aba9214fe16d17396ded8d715fc396a784fec"} err="failed to get container status \"72852594179db58c94d0cc967e6aba9214fe16d17396ded8d715fc396a784fec\": rpc error: code = NotFound desc = could not find container \"72852594179db58c94d0cc967e6aba9214fe16d17396ded8d715fc396a784fec\": container with ID starting with 72852594179db58c94d0cc967e6aba9214fe16d17396ded8d715fc396a784fec not found: ID does not exist" Jul 07 15:12:28.005735 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:28.005697 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-dad003-869564fdcb-5984r"] Jul 07 15:12:28.010028 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:28.010000 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-a-dad003-869564fdcb-5984r"] Jul 07 15:12:28.918944 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:28.918900 2555 patch_prober.go:28] interesting pod/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 07 15:12:28.919355 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:28.918974 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" podUID="009e99b4-b67a-472e-b8a2-4536d9d59bc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 07 15:12:29.043112 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:29.043074 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f1c3d81-9c34-4845-a386-f3bac463b191" path="/var/lib/kubelet/pods/9f1c3d81-9c34-4845-a386-f3bac463b191/volumes" Jul 07 15:12:29.994847 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:29.994810 2555 generic.go:356] "Generic (PLEG): container finished" podID="009e99b4-b67a-472e-b8a2-4536d9d59bc1" containerID="40410464836dfb0f1c4b10a83027efb8fddf6bc8475fc6ff0830af98a4607627" exitCode=0 Jul 07 15:12:29.995399 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:29.994867 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" event={"ID":"009e99b4-b67a-472e-b8a2-4536d9d59bc1","Type":"ContainerDied","Data":"40410464836dfb0f1c4b10a83027efb8fddf6bc8475fc6ff0830af98a4607627"} Jul 07 15:12:29.995399 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:29.995363 2555 scope.go:117] "RemoveContainer" containerID="40410464836dfb0f1c4b10a83027efb8fddf6bc8475fc6ff0830af98a4607627" Jul 07 15:12:29.996564 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:29.996535 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:12:29.996627 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:29.996605 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:12:29.996663 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:29.996639 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:12:31.001670 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:31.001632 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" event={"ID":"009e99b4-b67a-472e-b8a2-4536d9d59bc1","Type":"ContainerStarted","Data":"e7e6549b3697691fa09ee675f261134246510bfc065c61b538b93342c01e6f8e"} Jul 07 15:12:31.388710 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:31.388671 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" Jul 07 15:12:31.388974 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:31.388953 2555 patch_prober.go:28] interesting pod/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 07 15:12:31.389021 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:31.389001 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" podUID="009e99b4-b67a-472e-b8a2-4536d9d59bc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 07 15:12:32.389228 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:32.389191 2555 patch_prober.go:28] interesting pod/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 07 15:12:32.389652 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:32.389248 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" podUID="009e99b4-b67a-472e-b8a2-4536d9d59bc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 07 15:12:33.389675 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:33.389635 2555 patch_prober.go:28] interesting pod/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 07 15:12:33.390132 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:33.389697 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" podUID="009e99b4-b67a-472e-b8a2-4536d9d59bc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 07 15:12:33.640411 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:33.640285 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn"] Jul 07 15:12:33.645104 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:33.645057 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" podUID="da9f0899-8b8a-4e2d-9596-5762553020ee" containerName="istio-proxy" containerID="cri-o://f650896ecbcd58b7a4ee5ff431dbd8eb2a89841c21cad1664a543dcc2e687c5f" gracePeriod=30 Jul 07 15:12:34.388981 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:34.388945 2555 patch_prober.go:28] interesting pod/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 07 15:12:34.389178 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:34.389016 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" podUID="009e99b4-b67a-472e-b8a2-4536d9d59bc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 07 15:12:34.887843 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:34.887809 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-f21b85-65786b477f-tq9qc"] Jul 07 15:12:34.888277 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:34.888033 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-b-f21b85-65786b477f-tq9qc" podUID="e9b2da02-3359-41a4-8a05-2ff040be5b5e" containerName="maas-api" containerID="cri-o://50758f9096939f4810bfc130d0e2e4cd8fd048b3f75eea62618c8713c62c1054" gracePeriod=30 Jul 07 15:12:35.024416 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:35.024381 2555 generic.go:356] "Generic (PLEG): container finished" podID="e9b2da02-3359-41a4-8a05-2ff040be5b5e" containerID="50758f9096939f4810bfc130d0e2e4cd8fd048b3f75eea62618c8713c62c1054" exitCode=0 Jul 07 15:12:35.024603 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:35.024460 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-f21b85-65786b477f-tq9qc" event={"ID":"e9b2da02-3359-41a4-8a05-2ff040be5b5e","Type":"ContainerDied","Data":"50758f9096939f4810bfc130d0e2e4cd8fd048b3f75eea62618c8713c62c1054"} Jul 07 15:12:35.146090 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:35.146014 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-f21b85-65786b477f-tq9qc" Jul 07 15:12:35.304910 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:35.304856 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9b2da02-3359-41a4-8a05-2ff040be5b5e-maas-api-tls\") pod \"e9b2da02-3359-41a4-8a05-2ff040be5b5e\" (UID: \"e9b2da02-3359-41a4-8a05-2ff040be5b5e\") " Jul 07 15:12:35.305121 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:35.304960 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjbw8\" (UniqueName: \"kubernetes.io/projected/e9b2da02-3359-41a4-8a05-2ff040be5b5e-kube-api-access-mjbw8\") pod \"e9b2da02-3359-41a4-8a05-2ff040be5b5e\" (UID: \"e9b2da02-3359-41a4-8a05-2ff040be5b5e\") " Jul 07 15:12:35.305121 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:35.305014 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9b2da02-3359-41a4-8a05-2ff040be5b5e-openshift-service-ca\") pod \"e9b2da02-3359-41a4-8a05-2ff040be5b5e\" (UID: \"e9b2da02-3359-41a4-8a05-2ff040be5b5e\") " Jul 07 15:12:35.305414 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:35.305390 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9b2da02-3359-41a4-8a05-2ff040be5b5e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e9b2da02-3359-41a4-8a05-2ff040be5b5e" (UID: "e9b2da02-3359-41a4-8a05-2ff040be5b5e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:12:35.307176 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:35.307151 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9b2da02-3359-41a4-8a05-2ff040be5b5e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e9b2da02-3359-41a4-8a05-2ff040be5b5e" (UID: "e9b2da02-3359-41a4-8a05-2ff040be5b5e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:12:35.307289 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:35.307185 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9b2da02-3359-41a4-8a05-2ff040be5b5e-kube-api-access-mjbw8" (OuterVolumeSpecName: "kube-api-access-mjbw8") pod "e9b2da02-3359-41a4-8a05-2ff040be5b5e" (UID: "e9b2da02-3359-41a4-8a05-2ff040be5b5e"). InnerVolumeSpecName "kube-api-access-mjbw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:12:35.392227 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:35.392197 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" Jul 07 15:12:35.392593 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:35.392573 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" Jul 07 15:12:35.393169 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:35.393149 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" Jul 07 15:12:35.406799 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:35.406658 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9b2da02-3359-41a4-8a05-2ff040be5b5e-maas-api-tls\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:12:35.406799 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:35.406702 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjbw8\" (UniqueName: \"kubernetes.io/projected/e9b2da02-3359-41a4-8a05-2ff040be5b5e-kube-api-access-mjbw8\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:12:35.406799 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:35.406718 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9b2da02-3359-41a4-8a05-2ff040be5b5e-openshift-service-ca\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:12:36.030767 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:36.030728 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-f21b85-65786b477f-tq9qc" Jul 07 15:12:36.031248 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:36.030783 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-f21b85-65786b477f-tq9qc" event={"ID":"e9b2da02-3359-41a4-8a05-2ff040be5b5e","Type":"ContainerDied","Data":"7102f42227883dc8a0392d93e98820ebfbd6ddd71cc428c6efeba2d4990a238e"} Jul 07 15:12:36.031248 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:36.030846 2555 scope.go:117] "RemoveContainer" containerID="50758f9096939f4810bfc130d0e2e4cd8fd048b3f75eea62618c8713c62c1054" Jul 07 15:12:36.068473 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:36.068435 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-f21b85-65786b477f-tq9qc"] Jul 07 15:12:36.077548 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:36.077514 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-b-f21b85-65786b477f-tq9qc"] Jul 07 15:12:37.042170 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:37.042136 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9b2da02-3359-41a4-8a05-2ff040be5b5e" path="/var/lib/kubelet/pods/e9b2da02-3359-41a4-8a05-2ff040be5b5e/volumes" Jul 07 15:12:37.713667 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:37.713628 2555 patch_prober.go:28] interesting pod/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 07 15:12:37.713843 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:37.713691 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" podUID="da9f0899-8b8a-4e2d-9596-5762553020ee" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 15:12:38.915521 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:38.915490 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" Jul 07 15:12:39.043691 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:39.043609 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da9f0899-8b8a-4e2d-9596-5762553020ee-workload-certs\") pod \"da9f0899-8b8a-4e2d-9596-5762553020ee\" (UID: \"da9f0899-8b8a-4e2d-9596-5762553020ee\") " Jul 07 15:12:39.043880 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:39.043706 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da9f0899-8b8a-4e2d-9596-5762553020ee-istiod-ca-cert\") pod \"da9f0899-8b8a-4e2d-9596-5762553020ee\" (UID: \"da9f0899-8b8a-4e2d-9596-5762553020ee\") " Jul 07 15:12:39.043880 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:39.043732 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da9f0899-8b8a-4e2d-9596-5762553020ee-istio-podinfo\") pod \"da9f0899-8b8a-4e2d-9596-5762553020ee\" (UID: \"da9f0899-8b8a-4e2d-9596-5762553020ee\") " Jul 07 15:12:39.043880 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:39.043781 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da9f0899-8b8a-4e2d-9596-5762553020ee-istio-data\") pod \"da9f0899-8b8a-4e2d-9596-5762553020ee\" (UID: \"da9f0899-8b8a-4e2d-9596-5762553020ee\") " Jul 07 15:12:39.043880 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:39.043842 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da9f0899-8b8a-4e2d-9596-5762553020ee-istio-token\") pod \"da9f0899-8b8a-4e2d-9596-5762553020ee\" (UID: \"da9f0899-8b8a-4e2d-9596-5762553020ee\") " Jul 07 15:12:39.043880 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:39.043865 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da9f0899-8b8a-4e2d-9596-5762553020ee-istio-envoy\") pod \"da9f0899-8b8a-4e2d-9596-5762553020ee\" (UID: \"da9f0899-8b8a-4e2d-9596-5762553020ee\") " Jul 07 15:12:39.044163 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:39.043908 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bk22w\" (UniqueName: \"kubernetes.io/projected/da9f0899-8b8a-4e2d-9596-5762553020ee-kube-api-access-bk22w\") pod \"da9f0899-8b8a-4e2d-9596-5762553020ee\" (UID: \"da9f0899-8b8a-4e2d-9596-5762553020ee\") " Jul 07 15:12:39.044163 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:39.043945 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9f0899-8b8a-4e2d-9596-5762553020ee-credential-socket\") pod \"da9f0899-8b8a-4e2d-9596-5762553020ee\" (UID: \"da9f0899-8b8a-4e2d-9596-5762553020ee\") " Jul 07 15:12:39.044163 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:39.043943 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da9f0899-8b8a-4e2d-9596-5762553020ee-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "da9f0899-8b8a-4e2d-9596-5762553020ee" (UID: "da9f0899-8b8a-4e2d-9596-5762553020ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:12:39.044163 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:39.043970 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9f0899-8b8a-4e2d-9596-5762553020ee-workload-socket\") pod \"da9f0899-8b8a-4e2d-9596-5762553020ee\" (UID: \"da9f0899-8b8a-4e2d-9596-5762553020ee\") " Jul 07 15:12:39.044163 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:39.044152 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da9f0899-8b8a-4e2d-9596-5762553020ee-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "da9f0899-8b8a-4e2d-9596-5762553020ee" (UID: "da9f0899-8b8a-4e2d-9596-5762553020ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:12:39.044419 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:39.044279 2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da9f0899-8b8a-4e2d-9596-5762553020ee-istiod-ca-cert\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:12:39.044419 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:39.044294 2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da9f0899-8b8a-4e2d-9596-5762553020ee-workload-certs\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:12:39.044597 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:39.044573 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da9f0899-8b8a-4e2d-9596-5762553020ee-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "da9f0899-8b8a-4e2d-9596-5762553020ee" (UID: "da9f0899-8b8a-4e2d-9596-5762553020ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:12:39.044669 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:39.044615 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da9f0899-8b8a-4e2d-9596-5762553020ee-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "da9f0899-8b8a-4e2d-9596-5762553020ee" (UID: "da9f0899-8b8a-4e2d-9596-5762553020ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:12:39.044838 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:39.044811 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da9f0899-8b8a-4e2d-9596-5762553020ee-istio-data" (OuterVolumeSpecName: "istio-data") pod "da9f0899-8b8a-4e2d-9596-5762553020ee" (UID: "da9f0899-8b8a-4e2d-9596-5762553020ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:12:39.046369 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:39.046342 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da9f0899-8b8a-4e2d-9596-5762553020ee-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "da9f0899-8b8a-4e2d-9596-5762553020ee" (UID: "da9f0899-8b8a-4e2d-9596-5762553020ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 15:12:39.046489 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:39.046471 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da9f0899-8b8a-4e2d-9596-5762553020ee-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "da9f0899-8b8a-4e2d-9596-5762553020ee" (UID: "da9f0899-8b8a-4e2d-9596-5762553020ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:12:39.046862 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:39.046840 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da9f0899-8b8a-4e2d-9596-5762553020ee-istio-token" (OuterVolumeSpecName: "istio-token") pod "da9f0899-8b8a-4e2d-9596-5762553020ee" (UID: "da9f0899-8b8a-4e2d-9596-5762553020ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:12:39.046940 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:39.046856 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da9f0899-8b8a-4e2d-9596-5762553020ee-kube-api-access-bk22w" (OuterVolumeSpecName: "kube-api-access-bk22w") pod "da9f0899-8b8a-4e2d-9596-5762553020ee" (UID: "da9f0899-8b8a-4e2d-9596-5762553020ee"). InnerVolumeSpecName "kube-api-access-bk22w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:12:39.049263 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:39.049240 2555 generic.go:356] "Generic (PLEG): container finished" podID="da9f0899-8b8a-4e2d-9596-5762553020ee" containerID="f650896ecbcd58b7a4ee5ff431dbd8eb2a89841c21cad1664a543dcc2e687c5f" exitCode=0 Jul 07 15:12:39.049383 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:39.049287 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" event={"ID":"da9f0899-8b8a-4e2d-9596-5762553020ee","Type":"ContainerDied","Data":"f650896ecbcd58b7a4ee5ff431dbd8eb2a89841c21cad1664a543dcc2e687c5f"} Jul 07 15:12:39.049383 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:39.049302 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" Jul 07 15:12:39.049383 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:39.049329 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn" event={"ID":"da9f0899-8b8a-4e2d-9596-5762553020ee","Type":"ContainerDied","Data":"a3122c4c19cea1d72210bbcefb3cdc4129b66531127129efdff7558fa5225691"} Jul 07 15:12:39.049383 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:39.049350 2555 scope.go:117] "RemoveContainer" containerID="f650896ecbcd58b7a4ee5ff431dbd8eb2a89841c21cad1664a543dcc2e687c5f" Jul 07 15:12:39.067598 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:39.067575 2555 scope.go:117] "RemoveContainer" containerID="02eb27d0334b58cb61e399619caf4aae21dece01c8e7dd58fa6a199f145a5609" Jul 07 15:12:39.076941 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:39.076007 2555 scope.go:117] "RemoveContainer" containerID="f650896ecbcd58b7a4ee5ff431dbd8eb2a89841c21cad1664a543dcc2e687c5f" Jul 07 15:12:39.077079 ip-10-0-136-19 kubenswrapper[2555]: E0707 15:12:39.077055 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f650896ecbcd58b7a4ee5ff431dbd8eb2a89841c21cad1664a543dcc2e687c5f\": container with ID starting with f650896ecbcd58b7a4ee5ff431dbd8eb2a89841c21cad1664a543dcc2e687c5f not found: ID does not exist" containerID="f650896ecbcd58b7a4ee5ff431dbd8eb2a89841c21cad1664a543dcc2e687c5f" Jul 07 15:12:39.077146 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:39.077093 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f650896ecbcd58b7a4ee5ff431dbd8eb2a89841c21cad1664a543dcc2e687c5f"} err="failed to get container status \"f650896ecbcd58b7a4ee5ff431dbd8eb2a89841c21cad1664a543dcc2e687c5f\": rpc error: code = NotFound desc = could not find container \"f650896ecbcd58b7a4ee5ff431dbd8eb2a89841c21cad1664a543dcc2e687c5f\": container with ID starting with f650896ecbcd58b7a4ee5ff431dbd8eb2a89841c21cad1664a543dcc2e687c5f not found: ID does not exist" Jul 07 15:12:39.077146 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:39.077121 2555 scope.go:117] "RemoveContainer" containerID="02eb27d0334b58cb61e399619caf4aae21dece01c8e7dd58fa6a199f145a5609" Jul 07 15:12:39.077432 ip-10-0-136-19 kubenswrapper[2555]: E0707 15:12:39.077407 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02eb27d0334b58cb61e399619caf4aae21dece01c8e7dd58fa6a199f145a5609\": container with ID starting with 02eb27d0334b58cb61e399619caf4aae21dece01c8e7dd58fa6a199f145a5609 not found: ID does not exist" containerID="02eb27d0334b58cb61e399619caf4aae21dece01c8e7dd58fa6a199f145a5609" Jul 07 15:12:39.077506 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:39.077442 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02eb27d0334b58cb61e399619caf4aae21dece01c8e7dd58fa6a199f145a5609"} err="failed to get container status \"02eb27d0334b58cb61e399619caf4aae21dece01c8e7dd58fa6a199f145a5609\": rpc error: code = NotFound desc = could not find container \"02eb27d0334b58cb61e399619caf4aae21dece01c8e7dd58fa6a199f145a5609\": container with ID starting with 02eb27d0334b58cb61e399619caf4aae21dece01c8e7dd58fa6a199f145a5609 not found: ID does not exist" Jul 07 15:12:39.078628 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:39.078609 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn"] Jul 07 15:12:39.085301 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:39.085277 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-dad003-openshift-default-6c68bb4fb7-pcvmn"] Jul 07 15:12:39.145866 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:39.145820 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da9f0899-8b8a-4e2d-9596-5762553020ee-istio-token\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:12:39.145866 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:39.145860 2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da9f0899-8b8a-4e2d-9596-5762553020ee-istio-envoy\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:12:39.145866 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:39.145871 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk22w\" (UniqueName: \"kubernetes.io/projected/da9f0899-8b8a-4e2d-9596-5762553020ee-kube-api-access-bk22w\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:12:39.145866 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:39.145881 2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9f0899-8b8a-4e2d-9596-5762553020ee-credential-socket\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:12:39.146133 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:39.145890 2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9f0899-8b8a-4e2d-9596-5762553020ee-workload-socket\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:12:39.146133 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:39.145899 2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da9f0899-8b8a-4e2d-9596-5762553020ee-istio-podinfo\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:12:39.146133 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:39.145917 2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da9f0899-8b8a-4e2d-9596-5762553020ee-istio-data\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:12:40.814548 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:40.814515 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45"] Jul 07 15:12:40.814999 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:40.814706 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" podUID="009e99b4-b67a-472e-b8a2-4536d9d59bc1" containerName="istio-proxy" containerID="cri-o://e7e6549b3697691fa09ee675f261134246510bfc065c61b538b93342c01e6f8e" gracePeriod=30 Jul 07 15:12:41.042680 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:41.042635 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da9f0899-8b8a-4e2d-9596-5762553020ee" path="/var/lib/kubelet/pods/da9f0899-8b8a-4e2d-9596-5762553020ee/volumes" Jul 07 15:12:45.891666 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:45.891625 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-a-96d90d-6bf6b5c689-b748f"] Jul 07 15:12:45.892167 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:45.892128 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da9f0899-8b8a-4e2d-9596-5762553020ee" containerName="istio-proxy" Jul 07 15:12:45.892167 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:45.892146 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="da9f0899-8b8a-4e2d-9596-5762553020ee" containerName="istio-proxy" Jul 07 15:12:45.892167 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:45.892161 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9b2da02-3359-41a4-8a05-2ff040be5b5e" containerName="maas-api" Jul 07 15:12:45.892284 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:45.892169 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9b2da02-3359-41a4-8a05-2ff040be5b5e" containerName="maas-api" Jul 07 15:12:45.892284 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:45.892180 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f1c3d81-9c34-4845-a386-f3bac463b191" containerName="maas-api" Jul 07 15:12:45.892284 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:45.892187 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f1c3d81-9c34-4845-a386-f3bac463b191" containerName="maas-api" Jul 07 15:12:45.892284 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:45.892281 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="da9f0899-8b8a-4e2d-9596-5762553020ee" containerName="istio-proxy" Jul 07 15:12:45.892459 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:45.892292 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f1c3d81-9c34-4845-a386-f3bac463b191" containerName="maas-api" Jul 07 15:12:45.892459 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:45.892305 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9b2da02-3359-41a4-8a05-2ff040be5b5e" containerName="maas-api" Jul 07 15:12:45.895279 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:45.895260 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-96d90d-6bf6b5c689-b748f" Jul 07 15:12:45.897865 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:45.897842 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-a-96d90d-serving-cert\"" Jul 07 15:12:45.907722 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:45.907700 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-96d90d-6bf6b5c689-b748f"] Jul 07 15:12:46.012725 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.012686 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbg6g\" (UniqueName: \"kubernetes.io/projected/5cd20114-e2cc-42cc-adc7-28c860b2a629-kube-api-access-bbg6g\") pod \"maas-api-e2e-shared-a-96d90d-6bf6b5c689-b748f\" (UID: \"5cd20114-e2cc-42cc-adc7-28c860b2a629\") " pod="opendatahub/maas-api-e2e-shared-a-96d90d-6bf6b5c689-b748f" Jul 07 15:12:46.012919 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.012773 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cd20114-e2cc-42cc-adc7-28c860b2a629-openshift-service-ca\") pod \"maas-api-e2e-shared-a-96d90d-6bf6b5c689-b748f\" (UID: \"5cd20114-e2cc-42cc-adc7-28c860b2a629\") " pod="opendatahub/maas-api-e2e-shared-a-96d90d-6bf6b5c689-b748f" Jul 07 15:12:46.012919 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.012832 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cd20114-e2cc-42cc-adc7-28c860b2a629-maas-api-tls\") pod \"maas-api-e2e-shared-a-96d90d-6bf6b5c689-b748f\" (UID: \"5cd20114-e2cc-42cc-adc7-28c860b2a629\") " pod="opendatahub/maas-api-e2e-shared-a-96d90d-6bf6b5c689-b748f" Jul 07 15:12:46.072902 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.072876 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" Jul 07 15:12:46.085156 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.085128 2555 generic.go:356] "Generic (PLEG): container finished" podID="009e99b4-b67a-472e-b8a2-4536d9d59bc1" containerID="e7e6549b3697691fa09ee675f261134246510bfc065c61b538b93342c01e6f8e" exitCode=0 Jul 07 15:12:46.085276 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.085176 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" Jul 07 15:12:46.085276 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.085185 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" event={"ID":"009e99b4-b67a-472e-b8a2-4536d9d59bc1","Type":"ContainerDied","Data":"e7e6549b3697691fa09ee675f261134246510bfc065c61b538b93342c01e6f8e"} Jul 07 15:12:46.085276 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.085227 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45" event={"ID":"009e99b4-b67a-472e-b8a2-4536d9d59bc1","Type":"ContainerDied","Data":"863243bf219b2c6a31bf128e1d9cc258c025bba15973cfe209b40915317f9706"} Jul 07 15:12:46.085276 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.085248 2555 scope.go:117] "RemoveContainer" containerID="e7e6549b3697691fa09ee675f261134246510bfc065c61b538b93342c01e6f8e" Jul 07 15:12:46.093371 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.093355 2555 scope.go:117] "RemoveContainer" containerID="40410464836dfb0f1c4b10a83027efb8fddf6bc8475fc6ff0830af98a4607627" Jul 07 15:12:46.101036 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.101007 2555 scope.go:117] "RemoveContainer" containerID="e7e6549b3697691fa09ee675f261134246510bfc065c61b538b93342c01e6f8e" Jul 07 15:12:46.101278 ip-10-0-136-19 kubenswrapper[2555]: E0707 15:12:46.101258 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7e6549b3697691fa09ee675f261134246510bfc065c61b538b93342c01e6f8e\": container with ID starting with e7e6549b3697691fa09ee675f261134246510bfc065c61b538b93342c01e6f8e not found: ID does not exist" containerID="e7e6549b3697691fa09ee675f261134246510bfc065c61b538b93342c01e6f8e" Jul 07 15:12:46.101333 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.101284 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7e6549b3697691fa09ee675f261134246510bfc065c61b538b93342c01e6f8e"} err="failed to get container status \"e7e6549b3697691fa09ee675f261134246510bfc065c61b538b93342c01e6f8e\": rpc error: code = NotFound desc = could not find container \"e7e6549b3697691fa09ee675f261134246510bfc065c61b538b93342c01e6f8e\": container with ID starting with e7e6549b3697691fa09ee675f261134246510bfc065c61b538b93342c01e6f8e not found: ID does not exist" Jul 07 15:12:46.101333 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.101304 2555 scope.go:117] "RemoveContainer" containerID="40410464836dfb0f1c4b10a83027efb8fddf6bc8475fc6ff0830af98a4607627" Jul 07 15:12:46.101518 ip-10-0-136-19 kubenswrapper[2555]: E0707 15:12:46.101500 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40410464836dfb0f1c4b10a83027efb8fddf6bc8475fc6ff0830af98a4607627\": container with ID starting with 40410464836dfb0f1c4b10a83027efb8fddf6bc8475fc6ff0830af98a4607627 not found: ID does not exist" containerID="40410464836dfb0f1c4b10a83027efb8fddf6bc8475fc6ff0830af98a4607627" Jul 07 15:12:46.101560 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.101524 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40410464836dfb0f1c4b10a83027efb8fddf6bc8475fc6ff0830af98a4607627"} err="failed to get container status \"40410464836dfb0f1c4b10a83027efb8fddf6bc8475fc6ff0830af98a4607627\": rpc error: code = NotFound desc = could not find container \"40410464836dfb0f1c4b10a83027efb8fddf6bc8475fc6ff0830af98a4607627\": container with ID starting with 40410464836dfb0f1c4b10a83027efb8fddf6bc8475fc6ff0830af98a4607627 not found: ID does not exist" Jul 07 15:12:46.114402 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.114352 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbg6g\" (UniqueName: \"kubernetes.io/projected/5cd20114-e2cc-42cc-adc7-28c860b2a629-kube-api-access-bbg6g\") pod \"maas-api-e2e-shared-a-96d90d-6bf6b5c689-b748f\" (UID: \"5cd20114-e2cc-42cc-adc7-28c860b2a629\") " pod="opendatahub/maas-api-e2e-shared-a-96d90d-6bf6b5c689-b748f" Jul 07 15:12:46.114542 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.114429 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cd20114-e2cc-42cc-adc7-28c860b2a629-openshift-service-ca\") pod \"maas-api-e2e-shared-a-96d90d-6bf6b5c689-b748f\" (UID: \"5cd20114-e2cc-42cc-adc7-28c860b2a629\") " pod="opendatahub/maas-api-e2e-shared-a-96d90d-6bf6b5c689-b748f" Jul 07 15:12:46.114542 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.114483 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cd20114-e2cc-42cc-adc7-28c860b2a629-maas-api-tls\") pod \"maas-api-e2e-shared-a-96d90d-6bf6b5c689-b748f\" (UID: \"5cd20114-e2cc-42cc-adc7-28c860b2a629\") " pod="opendatahub/maas-api-e2e-shared-a-96d90d-6bf6b5c689-b748f" Jul 07 15:12:46.114658 ip-10-0-136-19 kubenswrapper[2555]: E0707 15:12:46.114609 2555 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-a-96d90d-serving-cert: secret "maas-api-e2e-shared-a-96d90d-serving-cert" not found Jul 07 15:12:46.114715 ip-10-0-136-19 kubenswrapper[2555]: E0707 15:12:46.114675 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cd20114-e2cc-42cc-adc7-28c860b2a629-maas-api-tls podName:5cd20114-e2cc-42cc-adc7-28c860b2a629 nodeName:}" failed. No retries permitted until 2026-07-07 15:12:46.614656199 +0000 UTC m=+2308.038842656 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5cd20114-e2cc-42cc-adc7-28c860b2a629-maas-api-tls") pod "maas-api-e2e-shared-a-96d90d-6bf6b5c689-b748f" (UID: "5cd20114-e2cc-42cc-adc7-28c860b2a629") : secret "maas-api-e2e-shared-a-96d90d-serving-cert" not found Jul 07 15:12:46.115062 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.115045 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cd20114-e2cc-42cc-adc7-28c860b2a629-openshift-service-ca\") pod \"maas-api-e2e-shared-a-96d90d-6bf6b5c689-b748f\" (UID: \"5cd20114-e2cc-42cc-adc7-28c860b2a629\") " pod="opendatahub/maas-api-e2e-shared-a-96d90d-6bf6b5c689-b748f" Jul 07 15:12:46.136239 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.136208 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbg6g\" (UniqueName: \"kubernetes.io/projected/5cd20114-e2cc-42cc-adc7-28c860b2a629-kube-api-access-bbg6g\") pod \"maas-api-e2e-shared-a-96d90d-6bf6b5c689-b748f\" (UID: \"5cd20114-e2cc-42cc-adc7-28c860b2a629\") " pod="opendatahub/maas-api-e2e-shared-a-96d90d-6bf6b5c689-b748f" Jul 07 15:12:46.214822 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.214790 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nxgd\" (UniqueName: \"kubernetes.io/projected/009e99b4-b67a-472e-b8a2-4536d9d59bc1-kube-api-access-6nxgd\") pod \"009e99b4-b67a-472e-b8a2-4536d9d59bc1\" (UID: \"009e99b4-b67a-472e-b8a2-4536d9d59bc1\") " Jul 07 15:12:46.214822 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.214823 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/009e99b4-b67a-472e-b8a2-4536d9d59bc1-istio-podinfo\") pod \"009e99b4-b67a-472e-b8a2-4536d9d59bc1\" (UID: \"009e99b4-b67a-472e-b8a2-4536d9d59bc1\") " Jul 07 15:12:46.215017 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.214840 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/009e99b4-b67a-472e-b8a2-4536d9d59bc1-istio-data\") pod \"009e99b4-b67a-472e-b8a2-4536d9d59bc1\" (UID: \"009e99b4-b67a-472e-b8a2-4536d9d59bc1\") " Jul 07 15:12:46.215017 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.214889 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/009e99b4-b67a-472e-b8a2-4536d9d59bc1-workload-socket\") pod \"009e99b4-b67a-472e-b8a2-4536d9d59bc1\" (UID: \"009e99b4-b67a-472e-b8a2-4536d9d59bc1\") " Jul 07 15:12:46.215017 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.214905 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/009e99b4-b67a-472e-b8a2-4536d9d59bc1-istiod-ca-cert\") pod \"009e99b4-b67a-472e-b8a2-4536d9d59bc1\" (UID: \"009e99b4-b67a-472e-b8a2-4536d9d59bc1\") " Jul 07 15:12:46.215017 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.214926 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/009e99b4-b67a-472e-b8a2-4536d9d59bc1-workload-certs\") pod \"009e99b4-b67a-472e-b8a2-4536d9d59bc1\" (UID: \"009e99b4-b67a-472e-b8a2-4536d9d59bc1\") " Jul 07 15:12:46.215017 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.214952 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/009e99b4-b67a-472e-b8a2-4536d9d59bc1-istio-envoy\") pod \"009e99b4-b67a-472e-b8a2-4536d9d59bc1\" (UID: \"009e99b4-b67a-472e-b8a2-4536d9d59bc1\") " Jul 07 15:12:46.215017 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.214976 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/009e99b4-b67a-472e-b8a2-4536d9d59bc1-istio-token\") pod \"009e99b4-b67a-472e-b8a2-4536d9d59bc1\" (UID: \"009e99b4-b67a-472e-b8a2-4536d9d59bc1\") " Jul 07 15:12:46.215301 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.215021 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/009e99b4-b67a-472e-b8a2-4536d9d59bc1-credential-socket\") pod \"009e99b4-b67a-472e-b8a2-4536d9d59bc1\" (UID: \"009e99b4-b67a-472e-b8a2-4536d9d59bc1\") " Jul 07 15:12:46.215301 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.215243 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/009e99b4-b67a-472e-b8a2-4536d9d59bc1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "009e99b4-b67a-472e-b8a2-4536d9d59bc1" (UID: "009e99b4-b67a-472e-b8a2-4536d9d59bc1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:12:46.215301 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.215259 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/009e99b4-b67a-472e-b8a2-4536d9d59bc1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "009e99b4-b67a-472e-b8a2-4536d9d59bc1" (UID: "009e99b4-b67a-472e-b8a2-4536d9d59bc1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:12:46.215456 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.215336 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/009e99b4-b67a-472e-b8a2-4536d9d59bc1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "009e99b4-b67a-472e-b8a2-4536d9d59bc1" (UID: "009e99b4-b67a-472e-b8a2-4536d9d59bc1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:12:46.215512 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.215488 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/009e99b4-b67a-472e-b8a2-4536d9d59bc1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "009e99b4-b67a-472e-b8a2-4536d9d59bc1" (UID: "009e99b4-b67a-472e-b8a2-4536d9d59bc1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:12:46.215632 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.215389 2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/009e99b4-b67a-472e-b8a2-4536d9d59bc1-workload-socket\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:12:46.215632 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.215614 2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/009e99b4-b67a-472e-b8a2-4536d9d59bc1-workload-certs\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:12:46.216036 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.216018 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/009e99b4-b67a-472e-b8a2-4536d9d59bc1-istio-data" (OuterVolumeSpecName: "istio-data") pod "009e99b4-b67a-472e-b8a2-4536d9d59bc1" (UID: "009e99b4-b67a-472e-b8a2-4536d9d59bc1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:12:46.217241 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.217217 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/009e99b4-b67a-472e-b8a2-4536d9d59bc1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "009e99b4-b67a-472e-b8a2-4536d9d59bc1" (UID: "009e99b4-b67a-472e-b8a2-4536d9d59bc1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:12:46.217345 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.217328 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/009e99b4-b67a-472e-b8a2-4536d9d59bc1-istio-token" (OuterVolumeSpecName: "istio-token") pod "009e99b4-b67a-472e-b8a2-4536d9d59bc1" (UID: "009e99b4-b67a-472e-b8a2-4536d9d59bc1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:12:46.217497 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.217482 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/009e99b4-b67a-472e-b8a2-4536d9d59bc1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "009e99b4-b67a-472e-b8a2-4536d9d59bc1" (UID: "009e99b4-b67a-472e-b8a2-4536d9d59bc1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 15:12:46.217574 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.217528 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/009e99b4-b67a-472e-b8a2-4536d9d59bc1-kube-api-access-6nxgd" (OuterVolumeSpecName: "kube-api-access-6nxgd") pod "009e99b4-b67a-472e-b8a2-4536d9d59bc1" (UID: "009e99b4-b67a-472e-b8a2-4536d9d59bc1"). InnerVolumeSpecName "kube-api-access-6nxgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:12:46.316627 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.316593 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nxgd\" (UniqueName: \"kubernetes.io/projected/009e99b4-b67a-472e-b8a2-4536d9d59bc1-kube-api-access-6nxgd\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:12:46.316627 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.316619 2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/009e99b4-b67a-472e-b8a2-4536d9d59bc1-istio-podinfo\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:12:46.316627 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.316628 2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/009e99b4-b67a-472e-b8a2-4536d9d59bc1-istio-data\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:12:46.316871 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.316637 2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/009e99b4-b67a-472e-b8a2-4536d9d59bc1-istiod-ca-cert\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:12:46.316871 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.316645 2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/009e99b4-b67a-472e-b8a2-4536d9d59bc1-istio-envoy\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:12:46.316871 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.316653 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/009e99b4-b67a-472e-b8a2-4536d9d59bc1-istio-token\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:12:46.316871 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.316660 2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/009e99b4-b67a-472e-b8a2-4536d9d59bc1-credential-socket\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:12:46.437973 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.437944 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45"] Jul 07 15:12:46.452792 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.452768 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-f21b85-openshift-default-5898ccbcfc-mnh45"] Jul 07 15:12:46.619672 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.619637 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cd20114-e2cc-42cc-adc7-28c860b2a629-maas-api-tls\") pod \"maas-api-e2e-shared-a-96d90d-6bf6b5c689-b748f\" (UID: \"5cd20114-e2cc-42cc-adc7-28c860b2a629\") " pod="opendatahub/maas-api-e2e-shared-a-96d90d-6bf6b5c689-b748f" Jul 07 15:12:46.622034 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.622009 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cd20114-e2cc-42cc-adc7-28c860b2a629-maas-api-tls\") pod \"maas-api-e2e-shared-a-96d90d-6bf6b5c689-b748f\" (UID: \"5cd20114-e2cc-42cc-adc7-28c860b2a629\") " pod="opendatahub/maas-api-e2e-shared-a-96d90d-6bf6b5c689-b748f" Jul 07 15:12:46.819398 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.819325 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-96d90d-6bf6b5c689-b748f" Jul 07 15:12:46.964655 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:46.964632 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-96d90d-6bf6b5c689-b748f"] Jul 07 15:12:46.966794 ip-10-0-136-19 kubenswrapper[2555]: W0707 15:12:46.966766 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cd20114_e2cc_42cc_adc7_28c860b2a629.slice/crio-f8c6cc7795c78534b141fe2826572390dfece61c242cdb629f828003192c3143 WatchSource:0}: Error finding container f8c6cc7795c78534b141fe2826572390dfece61c242cdb629f828003192c3143: Status 404 returned error can't find the container with id f8c6cc7795c78534b141fe2826572390dfece61c242cdb629f828003192c3143 Jul 07 15:12:47.041419 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:47.041379 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="009e99b4-b67a-472e-b8a2-4536d9d59bc1" path="/var/lib/kubelet/pods/009e99b4-b67a-472e-b8a2-4536d9d59bc1/volumes" Jul 07 15:12:47.092313 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:47.092278 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-96d90d-6bf6b5c689-b748f" event={"ID":"5cd20114-e2cc-42cc-adc7-28c860b2a629","Type":"ContainerStarted","Data":"f8c6cc7795c78534b141fe2826572390dfece61c242cdb629f828003192c3143"} Jul 07 15:12:48.098354 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:48.098319 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-96d90d-6bf6b5c689-b748f" event={"ID":"5cd20114-e2cc-42cc-adc7-28c860b2a629","Type":"ContainerStarted","Data":"efc7ae9ba9fba2063d0ad9c95fdb19f7545ec90a6a04458818edccc31a77bc15"} Jul 07 15:12:48.098801 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:48.098514 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-a-96d90d-6bf6b5c689-b748f" Jul 07 15:12:48.116354 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:48.116304 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-a-96d90d-6bf6b5c689-b748f" podStartSLOduration=2.714921791 podStartE2EDuration="3.116288959s" totalImagesPullingTime="401.367168ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:12:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:12:48.115477623 +0000 UTC m=+2309.539664099" watchObservedRunningTime="2026-07-07 15:12:48.116288959 +0000 UTC m=+2309.540475433" Jul 07 15:12:51.164458 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:51.164428 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj"] Jul 07 15:12:51.165060 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:51.164821 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="009e99b4-b67a-472e-b8a2-4536d9d59bc1" containerName="istio-proxy" Jul 07 15:12:51.165060 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:51.164840 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="009e99b4-b67a-472e-b8a2-4536d9d59bc1" containerName="istio-proxy" Jul 07 15:12:51.165060 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:51.164864 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="009e99b4-b67a-472e-b8a2-4536d9d59bc1" containerName="istio-proxy" Jul 07 15:12:51.165060 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:51.164874 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="009e99b4-b67a-472e-b8a2-4536d9d59bc1" containerName="istio-proxy" Jul 07 15:12:51.165060 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:51.164900 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da9f0899-8b8a-4e2d-9596-5762553020ee" containerName="istio-proxy" Jul 07 15:12:51.165060 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:51.164910 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="da9f0899-8b8a-4e2d-9596-5762553020ee" containerName="istio-proxy" Jul 07 15:12:51.165060 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:51.165027 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="009e99b4-b67a-472e-b8a2-4536d9d59bc1" containerName="istio-proxy" Jul 07 15:12:51.165060 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:51.165042 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="da9f0899-8b8a-4e2d-9596-5762553020ee" containerName="istio-proxy" Jul 07 15:12:51.165060 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:51.165055 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="009e99b4-b67a-472e-b8a2-4536d9d59bc1" containerName="istio-proxy" Jul 07 15:12:51.168041 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:51.168022 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" Jul 07 15:12:51.170992 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:51.170969 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-17f250-openshift-default-dockercfg-hhcqt\"" Jul 07 15:12:51.183375 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:51.183352 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj"] Jul 07 15:12:51.363568 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:51.363538 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z8j8\" (UniqueName: \"kubernetes.io/projected/4b2d9292-e7b3-431a-8eab-ea585b8ad994-kube-api-access-8z8j8\") pod \"e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj\" (UID: \"4b2d9292-e7b3-431a-8eab-ea585b8ad994\") " pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" Jul 07 15:12:51.363738 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:51.363578 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b2d9292-e7b3-431a-8eab-ea585b8ad994-credential-socket\") pod \"e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj\" (UID: \"4b2d9292-e7b3-431a-8eab-ea585b8ad994\") " pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" Jul 07 15:12:51.363738 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:51.363599 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b2d9292-e7b3-431a-8eab-ea585b8ad994-istiod-ca-cert\") pod \"e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj\" (UID: \"4b2d9292-e7b3-431a-8eab-ea585b8ad994\") " pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" Jul 07 15:12:51.363738 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:51.363620 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b2d9292-e7b3-431a-8eab-ea585b8ad994-istio-envoy\") pod \"e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj\" (UID: \"4b2d9292-e7b3-431a-8eab-ea585b8ad994\") " pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" Jul 07 15:12:51.363738 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:51.363722 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b2d9292-e7b3-431a-8eab-ea585b8ad994-istio-token\") pod \"e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj\" (UID: \"4b2d9292-e7b3-431a-8eab-ea585b8ad994\") " pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" Jul 07 15:12:51.363901 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:51.363782 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b2d9292-e7b3-431a-8eab-ea585b8ad994-istio-data\") pod \"e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj\" (UID: \"4b2d9292-e7b3-431a-8eab-ea585b8ad994\") " pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" Jul 07 15:12:51.363901 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:51.363826 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b2d9292-e7b3-431a-8eab-ea585b8ad994-istio-podinfo\") pod \"e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj\" (UID: \"4b2d9292-e7b3-431a-8eab-ea585b8ad994\") " pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" Jul 07 15:12:51.363901 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:51.363879 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b2d9292-e7b3-431a-8eab-ea585b8ad994-workload-socket\") pod \"e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj\" (UID: \"4b2d9292-e7b3-431a-8eab-ea585b8ad994\") " pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" Jul 07 15:12:51.363999 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:51.363912 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b2d9292-e7b3-431a-8eab-ea585b8ad994-workload-certs\") pod \"e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj\" (UID: \"4b2d9292-e7b3-431a-8eab-ea585b8ad994\") " pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" Jul 07 15:12:51.465447 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:51.465363 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b2d9292-e7b3-431a-8eab-ea585b8ad994-istio-podinfo\") pod \"e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj\" (UID: \"4b2d9292-e7b3-431a-8eab-ea585b8ad994\") " pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" Jul 07 15:12:51.465447 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:51.465405 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b2d9292-e7b3-431a-8eab-ea585b8ad994-workload-socket\") pod \"e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj\" (UID: \"4b2d9292-e7b3-431a-8eab-ea585b8ad994\") " pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" Jul 07 15:12:51.465663 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:51.465526 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b2d9292-e7b3-431a-8eab-ea585b8ad994-workload-certs\") pod \"e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj\" (UID: \"4b2d9292-e7b3-431a-8eab-ea585b8ad994\") " pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" Jul 07 15:12:51.465663 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:51.465597 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z8j8\" (UniqueName: \"kubernetes.io/projected/4b2d9292-e7b3-431a-8eab-ea585b8ad994-kube-api-access-8z8j8\") pod \"e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj\" (UID: \"4b2d9292-e7b3-431a-8eab-ea585b8ad994\") " pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" Jul 07 15:12:51.465663 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:51.465655 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b2d9292-e7b3-431a-8eab-ea585b8ad994-credential-socket\") pod \"e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj\" (UID: \"4b2d9292-e7b3-431a-8eab-ea585b8ad994\") " pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" Jul 07 15:12:51.465870 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:51.465693 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b2d9292-e7b3-431a-8eab-ea585b8ad994-istiod-ca-cert\") pod \"e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj\" (UID: \"4b2d9292-e7b3-431a-8eab-ea585b8ad994\") " pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" Jul 07 15:12:51.465870 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:51.465773 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b2d9292-e7b3-431a-8eab-ea585b8ad994-workload-socket\") pod \"e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj\" (UID: \"4b2d9292-e7b3-431a-8eab-ea585b8ad994\") " pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" Jul 07 15:12:51.465870 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:51.465797 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b2d9292-e7b3-431a-8eab-ea585b8ad994-istio-envoy\") pod \"e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj\" (UID: \"4b2d9292-e7b3-431a-8eab-ea585b8ad994\") " pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" Jul 07 15:12:51.466048 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:51.465886 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b2d9292-e7b3-431a-8eab-ea585b8ad994-istio-token\") pod \"e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj\" (UID: \"4b2d9292-e7b3-431a-8eab-ea585b8ad994\") " pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" Jul 07 15:12:51.466048 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:51.465898 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b2d9292-e7b3-431a-8eab-ea585b8ad994-workload-certs\") pod \"e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj\" (UID: \"4b2d9292-e7b3-431a-8eab-ea585b8ad994\") " pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" Jul 07 15:12:51.466048 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:51.465919 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b2d9292-e7b3-431a-8eab-ea585b8ad994-istio-data\") pod \"e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj\" (UID: \"4b2d9292-e7b3-431a-8eab-ea585b8ad994\") " pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" Jul 07 15:12:51.466214 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:51.466115 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b2d9292-e7b3-431a-8eab-ea585b8ad994-credential-socket\") pod \"e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj\" (UID: \"4b2d9292-e7b3-431a-8eab-ea585b8ad994\") " pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" Jul 07 15:12:51.466214 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:51.466171 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b2d9292-e7b3-431a-8eab-ea585b8ad994-istio-data\") pod \"e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj\" (UID: \"4b2d9292-e7b3-431a-8eab-ea585b8ad994\") " pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" Jul 07 15:12:51.466320 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:51.466300 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b2d9292-e7b3-431a-8eab-ea585b8ad994-istiod-ca-cert\") pod \"e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj\" (UID: \"4b2d9292-e7b3-431a-8eab-ea585b8ad994\") " pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" Jul 07 15:12:51.468071 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:51.468051 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b2d9292-e7b3-431a-8eab-ea585b8ad994-istio-podinfo\") pod \"e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj\" (UID: \"4b2d9292-e7b3-431a-8eab-ea585b8ad994\") " pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" Jul 07 15:12:51.468173 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:51.468145 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b2d9292-e7b3-431a-8eab-ea585b8ad994-istio-envoy\") pod \"e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj\" (UID: \"4b2d9292-e7b3-431a-8eab-ea585b8ad994\") " pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" Jul 07 15:12:51.474696 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:51.474669 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b2d9292-e7b3-431a-8eab-ea585b8ad994-istio-token\") pod \"e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj\" (UID: \"4b2d9292-e7b3-431a-8eab-ea585b8ad994\") " pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" Jul 07 15:12:51.474829 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:51.474719 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z8j8\" (UniqueName: \"kubernetes.io/projected/4b2d9292-e7b3-431a-8eab-ea585b8ad994-kube-api-access-8z8j8\") pod \"e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj\" (UID: \"4b2d9292-e7b3-431a-8eab-ea585b8ad994\") " pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" Jul 07 15:12:51.478728 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:51.478706 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" Jul 07 15:12:51.619469 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:51.619442 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj"] Jul 07 15:12:51.621402 ip-10-0-136-19 kubenswrapper[2555]: W0707 15:12:51.621375 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b2d9292_e7b3_431a_8eab_ea585b8ad994.slice/crio-02fae47fab5e181435aefe8a900d693c2250b1951ed2004cd0b6a8f5f96db2f0 WatchSource:0}: Error finding container 02fae47fab5e181435aefe8a900d693c2250b1951ed2004cd0b6a8f5f96db2f0: Status 404 returned error can't find the container with id 02fae47fab5e181435aefe8a900d693c2250b1951ed2004cd0b6a8f5f96db2f0 Jul 07 15:12:51.624010 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:51.623975 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:12:51.624105 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:51.624044 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:12:51.624105 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:51.624074 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:12:52.119476 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:52.119439 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" event={"ID":"4b2d9292-e7b3-431a-8eab-ea585b8ad994","Type":"ContainerStarted","Data":"0c70972de439cc7ef006381c2719a0756a88c22ce0978f2f50a4749c349218fa"} Jul 07 15:12:52.119684 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:52.119494 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" event={"ID":"4b2d9292-e7b3-431a-8eab-ea585b8ad994","Type":"ContainerStarted","Data":"02fae47fab5e181435aefe8a900d693c2250b1951ed2004cd0b6a8f5f96db2f0"} Jul 07 15:12:52.148778 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:52.146866 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" podStartSLOduration=1.14683213 podStartE2EDuration="1.14683213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:12:52.143966322 +0000 UTC m=+2313.568152801" watchObservedRunningTime="2026-07-07 15:12:52.14683213 +0000 UTC m=+2313.571018600" Jul 07 15:12:52.479074 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:52.478993 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" Jul 07 15:12:52.484007 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:52.483980 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" Jul 07 15:12:53.123995 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:53.123963 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" Jul 07 15:12:53.124876 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:53.124857 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" Jul 07 15:12:54.109824 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:54.109796 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-a-96d90d-6bf6b5c689-b748f" Jul 07 15:12:54.328017 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:54.327986 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-b-17f250-667fd95f75-hcntc"] Jul 07 15:12:54.331566 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:54.331548 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-17f250-667fd95f75-hcntc" Jul 07 15:12:54.334488 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:54.334464 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-17f250-serving-cert\"" Jul 07 15:12:54.360843 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:54.360742 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-17f250-667fd95f75-hcntc"] Jul 07 15:12:54.393717 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:54.393676 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff015d55-c25e-4ae6-adbf-72882996c1a4-maas-api-tls\") pod \"maas-api-e2e-shared-b-17f250-667fd95f75-hcntc\" (UID: \"ff015d55-c25e-4ae6-adbf-72882996c1a4\") " pod="opendatahub/maas-api-e2e-shared-b-17f250-667fd95f75-hcntc" Jul 07 15:12:54.393717 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:54.393716 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff015d55-c25e-4ae6-adbf-72882996c1a4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-17f250-667fd95f75-hcntc\" (UID: \"ff015d55-c25e-4ae6-adbf-72882996c1a4\") " pod="opendatahub/maas-api-e2e-shared-b-17f250-667fd95f75-hcntc" Jul 07 15:12:54.393982 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:54.393794 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27pnl\" (UniqueName: \"kubernetes.io/projected/ff015d55-c25e-4ae6-adbf-72882996c1a4-kube-api-access-27pnl\") pod \"maas-api-e2e-shared-b-17f250-667fd95f75-hcntc\" (UID: \"ff015d55-c25e-4ae6-adbf-72882996c1a4\") " pod="opendatahub/maas-api-e2e-shared-b-17f250-667fd95f75-hcntc" Jul 07 15:12:54.495228 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:54.495185 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff015d55-c25e-4ae6-adbf-72882996c1a4-maas-api-tls\") pod \"maas-api-e2e-shared-b-17f250-667fd95f75-hcntc\" (UID: \"ff015d55-c25e-4ae6-adbf-72882996c1a4\") " pod="opendatahub/maas-api-e2e-shared-b-17f250-667fd95f75-hcntc" Jul 07 15:12:54.495228 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:54.495230 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff015d55-c25e-4ae6-adbf-72882996c1a4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-17f250-667fd95f75-hcntc\" (UID: \"ff015d55-c25e-4ae6-adbf-72882996c1a4\") " pod="opendatahub/maas-api-e2e-shared-b-17f250-667fd95f75-hcntc" Jul 07 15:12:54.495519 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:54.495253 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27pnl\" (UniqueName: \"kubernetes.io/projected/ff015d55-c25e-4ae6-adbf-72882996c1a4-kube-api-access-27pnl\") pod \"maas-api-e2e-shared-b-17f250-667fd95f75-hcntc\" (UID: \"ff015d55-c25e-4ae6-adbf-72882996c1a4\") " pod="opendatahub/maas-api-e2e-shared-b-17f250-667fd95f75-hcntc" Jul 07 15:12:54.495519 ip-10-0-136-19 kubenswrapper[2555]: E0707 15:12:54.495370 2555 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-b-17f250-serving-cert: secret "maas-api-e2e-shared-b-17f250-serving-cert" not found Jul 07 15:12:54.495519 ip-10-0-136-19 kubenswrapper[2555]: E0707 15:12:54.495463 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff015d55-c25e-4ae6-adbf-72882996c1a4-maas-api-tls podName:ff015d55-c25e-4ae6-adbf-72882996c1a4 nodeName:}" failed. No retries permitted until 2026-07-07 15:12:54.995440736 +0000 UTC m=+2316.419627205 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ff015d55-c25e-4ae6-adbf-72882996c1a4-maas-api-tls") pod "maas-api-e2e-shared-b-17f250-667fd95f75-hcntc" (UID: "ff015d55-c25e-4ae6-adbf-72882996c1a4") : secret "maas-api-e2e-shared-b-17f250-serving-cert" not found Jul 07 15:12:54.495956 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:54.495936 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff015d55-c25e-4ae6-adbf-72882996c1a4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-17f250-667fd95f75-hcntc\" (UID: \"ff015d55-c25e-4ae6-adbf-72882996c1a4\") " pod="opendatahub/maas-api-e2e-shared-b-17f250-667fd95f75-hcntc" Jul 07 15:12:54.507706 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:54.507680 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27pnl\" (UniqueName: \"kubernetes.io/projected/ff015d55-c25e-4ae6-adbf-72882996c1a4-kube-api-access-27pnl\") pod \"maas-api-e2e-shared-b-17f250-667fd95f75-hcntc\" (UID: \"ff015d55-c25e-4ae6-adbf-72882996c1a4\") " pod="opendatahub/maas-api-e2e-shared-b-17f250-667fd95f75-hcntc" Jul 07 15:12:55.000026 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:54.999991 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff015d55-c25e-4ae6-adbf-72882996c1a4-maas-api-tls\") pod \"maas-api-e2e-shared-b-17f250-667fd95f75-hcntc\" (UID: \"ff015d55-c25e-4ae6-adbf-72882996c1a4\") " pod="opendatahub/maas-api-e2e-shared-b-17f250-667fd95f75-hcntc" Jul 07 15:12:55.002502 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:55.002476 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff015d55-c25e-4ae6-adbf-72882996c1a4-maas-api-tls\") pod \"maas-api-e2e-shared-b-17f250-667fd95f75-hcntc\" (UID: \"ff015d55-c25e-4ae6-adbf-72882996c1a4\") " pod="opendatahub/maas-api-e2e-shared-b-17f250-667fd95f75-hcntc" Jul 07 15:12:55.242846 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:55.242738 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-17f250-667fd95f75-hcntc" Jul 07 15:12:55.377939 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:55.377889 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-17f250-667fd95f75-hcntc"] Jul 07 15:12:55.379958 ip-10-0-136-19 kubenswrapper[2555]: W0707 15:12:55.379933 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff015d55_c25e_4ae6_adbf_72882996c1a4.slice/crio-39c25c7a4dda361be06522ea424ce871f8bda2960ebf6aa7b33f74897f242998 WatchSource:0}: Error finding container 39c25c7a4dda361be06522ea424ce871f8bda2960ebf6aa7b33f74897f242998: Status 404 returned error can't find the container with id 39c25c7a4dda361be06522ea424ce871f8bda2960ebf6aa7b33f74897f242998 Jul 07 15:12:56.142929 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:56.142898 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-17f250-667fd95f75-hcntc" event={"ID":"ff015d55-c25e-4ae6-adbf-72882996c1a4","Type":"ContainerStarted","Data":"d07a21c8209cf4823e6513b33f973126eac6bcb962be1142aefca2ed2833fb66"} Jul 07 15:12:56.143099 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:56.142947 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-17f250-667fd95f75-hcntc" event={"ID":"ff015d55-c25e-4ae6-adbf-72882996c1a4","Type":"ContainerStarted","Data":"39c25c7a4dda361be06522ea424ce871f8bda2960ebf6aa7b33f74897f242998"} Jul 07 15:12:56.143099 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:56.142993 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-b-17f250-667fd95f75-hcntc" Jul 07 15:12:56.163460 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:12:56.163411 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-b-17f250-667fd95f75-hcntc" podStartSLOduration=1.7692483399999999 podStartE2EDuration="2.163397424s" totalImagesPullingTime="394.149084ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:12:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:12:56.161919612 +0000 UTC m=+2317.586106089" watchObservedRunningTime="2026-07-07 15:12:56.163397424 +0000 UTC m=+2317.587583899" Jul 07 15:13:02.153031 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:13:02.153001 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-b-17f250-667fd95f75-hcntc" Jul 07 15:13:09.125978 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:13:09.125933 2555 patch_prober.go:28] interesting pod/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 07 15:13:09.126383 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:13:09.126005 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" podUID="4b2d9292-e7b3-431a-8eab-ea585b8ad994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 07 15:13:10.212250 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:13:10.212216 2555 generic.go:356] "Generic (PLEG): container finished" podID="4b2d9292-e7b3-431a-8eab-ea585b8ad994" containerID="0c70972de439cc7ef006381c2719a0756a88c22ce0978f2f50a4749c349218fa" exitCode=0 Jul 07 15:13:10.212674 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:13:10.212289 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" event={"ID":"4b2d9292-e7b3-431a-8eab-ea585b8ad994","Type":"ContainerDied","Data":"0c70972de439cc7ef006381c2719a0756a88c22ce0978f2f50a4749c349218fa"} Jul 07 15:13:10.212774 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:13:10.212735 2555 scope.go:117] "RemoveContainer" containerID="0c70972de439cc7ef006381c2719a0756a88c22ce0978f2f50a4749c349218fa" Jul 07 15:13:10.213812 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:13:10.213785 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:13:10.213877 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:13:10.213864 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:13:10.213927 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:13:10.213899 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:13:11.218596 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:13:11.218513 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" event={"ID":"4b2d9292-e7b3-431a-8eab-ea585b8ad994","Type":"ContainerStarted","Data":"153e7014a018837ee4c469e90f3ef04e8ec918f50ae0840ee6c0e6ba786d5614"} Jul 07 15:13:11.480001 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:13:11.479906 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" Jul 07 15:13:11.480166 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:13:11.480123 2555 patch_prober.go:28] interesting pod/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 07 15:13:11.480217 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:13:11.480170 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" podUID="4b2d9292-e7b3-431a-8eab-ea585b8ad994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 07 15:13:12.479857 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:13:12.479823 2555 patch_prober.go:28] interesting pod/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 07 15:13:12.480333 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:13:12.479887 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" podUID="4b2d9292-e7b3-431a-8eab-ea585b8ad994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 07 15:13:13.480151 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:13:13.480112 2555 patch_prober.go:28] interesting pod/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 07 15:13:13.480585 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:13:13.480177 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" podUID="4b2d9292-e7b3-431a-8eab-ea585b8ad994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 07 15:13:14.479384 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:13:14.479345 2555 patch_prober.go:28] interesting pod/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 07 15:13:14.479609 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:13:14.479413 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" podUID="4b2d9292-e7b3-431a-8eab-ea585b8ad994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 07 15:13:15.479282 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:13:15.479248 2555 patch_prober.go:28] interesting pod/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 07 15:13:15.479701 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:13:15.479304 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" podUID="4b2d9292-e7b3-431a-8eab-ea585b8ad994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 07 15:13:16.483302 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:13:16.483268 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" Jul 07 15:13:16.483786 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:13:16.483650 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" Jul 07 15:13:16.484442 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:13:16.484426 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" Jul 07 15:18:14.070397 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:14.070364 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-96d90d-6bf6b5c689-b748f"] Jul 07 15:18:14.072911 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:14.070587 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-a-96d90d-6bf6b5c689-b748f" podUID="5cd20114-e2cc-42cc-adc7-28c860b2a629" containerName="maas-api" containerID="cri-o://efc7ae9ba9fba2063d0ad9c95fdb19f7545ec90a6a04458818edccc31a77bc15" gracePeriod=30 Jul 07 15:18:14.104253 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:14.104220 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-api-e2e-shared-a-96d90d-6bf6b5c689-b748f" podUID="5cd20114-e2cc-42cc-adc7-28c860b2a629" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.51:8443/health\": dial tcp 10.133.0.51:8443: connect: connection refused" Jul 07 15:18:14.319992 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:14.319968 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-96d90d-6bf6b5c689-b748f" Jul 07 15:18:14.423040 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:14.423010 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbg6g\" (UniqueName: \"kubernetes.io/projected/5cd20114-e2cc-42cc-adc7-28c860b2a629-kube-api-access-bbg6g\") pod \"5cd20114-e2cc-42cc-adc7-28c860b2a629\" (UID: \"5cd20114-e2cc-42cc-adc7-28c860b2a629\") " Jul 07 15:18:14.423245 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:14.423073 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cd20114-e2cc-42cc-adc7-28c860b2a629-maas-api-tls\") pod \"5cd20114-e2cc-42cc-adc7-28c860b2a629\" (UID: \"5cd20114-e2cc-42cc-adc7-28c860b2a629\") " Jul 07 15:18:14.423245 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:14.423122 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cd20114-e2cc-42cc-adc7-28c860b2a629-openshift-service-ca\") pod \"5cd20114-e2cc-42cc-adc7-28c860b2a629\" (UID: \"5cd20114-e2cc-42cc-adc7-28c860b2a629\") " Jul 07 15:18:14.423472 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:14.423447 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cd20114-e2cc-42cc-adc7-28c860b2a629-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5cd20114-e2cc-42cc-adc7-28c860b2a629" (UID: "5cd20114-e2cc-42cc-adc7-28c860b2a629"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:18:14.425087 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:14.425057 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cd20114-e2cc-42cc-adc7-28c860b2a629-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5cd20114-e2cc-42cc-adc7-28c860b2a629" (UID: "5cd20114-e2cc-42cc-adc7-28c860b2a629"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:18:14.425178 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:14.425162 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cd20114-e2cc-42cc-adc7-28c860b2a629-kube-api-access-bbg6g" (OuterVolumeSpecName: "kube-api-access-bbg6g") pod "5cd20114-e2cc-42cc-adc7-28c860b2a629" (UID: "5cd20114-e2cc-42cc-adc7-28c860b2a629"). InnerVolumeSpecName "kube-api-access-bbg6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:18:14.524002 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:14.523968 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cd20114-e2cc-42cc-adc7-28c860b2a629-openshift-service-ca\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:18:14.524002 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:14.523996 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbg6g\" (UniqueName: \"kubernetes.io/projected/5cd20114-e2cc-42cc-adc7-28c860b2a629-kube-api-access-bbg6g\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:18:14.524180 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:14.524013 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cd20114-e2cc-42cc-adc7-28c860b2a629-maas-api-tls\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:18:14.676958 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:14.676867 2555 generic.go:356] "Generic (PLEG): container finished" podID="5cd20114-e2cc-42cc-adc7-28c860b2a629" containerID="efc7ae9ba9fba2063d0ad9c95fdb19f7545ec90a6a04458818edccc31a77bc15" exitCode=0 Jul 07 15:18:14.676958 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:14.676912 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-96d90d-6bf6b5c689-b748f" Jul 07 15:18:14.676958 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:14.676946 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-96d90d-6bf6b5c689-b748f" event={"ID":"5cd20114-e2cc-42cc-adc7-28c860b2a629","Type":"ContainerDied","Data":"efc7ae9ba9fba2063d0ad9c95fdb19f7545ec90a6a04458818edccc31a77bc15"} Jul 07 15:18:14.677200 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:14.677001 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-96d90d-6bf6b5c689-b748f" event={"ID":"5cd20114-e2cc-42cc-adc7-28c860b2a629","Type":"ContainerDied","Data":"f8c6cc7795c78534b141fe2826572390dfece61c242cdb629f828003192c3143"} Jul 07 15:18:14.677200 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:14.677031 2555 scope.go:117] "RemoveContainer" containerID="efc7ae9ba9fba2063d0ad9c95fdb19f7545ec90a6a04458818edccc31a77bc15" Jul 07 15:18:14.686572 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:14.686555 2555 scope.go:117] "RemoveContainer" containerID="efc7ae9ba9fba2063d0ad9c95fdb19f7545ec90a6a04458818edccc31a77bc15" Jul 07 15:18:14.686856 ip-10-0-136-19 kubenswrapper[2555]: E0707 15:18:14.686836 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efc7ae9ba9fba2063d0ad9c95fdb19f7545ec90a6a04458818edccc31a77bc15\": container with ID starting with efc7ae9ba9fba2063d0ad9c95fdb19f7545ec90a6a04458818edccc31a77bc15 not found: ID does not exist" containerID="efc7ae9ba9fba2063d0ad9c95fdb19f7545ec90a6a04458818edccc31a77bc15" Jul 07 15:18:14.686930 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:14.686862 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efc7ae9ba9fba2063d0ad9c95fdb19f7545ec90a6a04458818edccc31a77bc15"} err="failed to get container status \"efc7ae9ba9fba2063d0ad9c95fdb19f7545ec90a6a04458818edccc31a77bc15\": rpc error: code = NotFound desc = could not find container \"efc7ae9ba9fba2063d0ad9c95fdb19f7545ec90a6a04458818edccc31a77bc15\": container with ID starting with efc7ae9ba9fba2063d0ad9c95fdb19f7545ec90a6a04458818edccc31a77bc15 not found: ID does not exist" Jul 07 15:18:14.699149 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:14.699123 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-96d90d-6bf6b5c689-b748f"] Jul 07 15:18:14.701614 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:14.701593 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-96d90d-6bf6b5c689-b748f"] Jul 07 15:18:15.042343 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:15.042265 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cd20114-e2cc-42cc-adc7-28c860b2a629" path="/var/lib/kubelet/pods/5cd20114-e2cc-42cc-adc7-28c860b2a629/volumes" Jul 07 15:18:57.025766 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:57.025717 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-17f250-667fd95f75-hcntc"] Jul 07 15:18:57.026433 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:57.026399 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-b-17f250-667fd95f75-hcntc" podUID="ff015d55-c25e-4ae6-adbf-72882996c1a4" containerName="maas-api" containerID="cri-o://d07a21c8209cf4823e6513b33f973126eac6bcb962be1142aefca2ed2833fb66" gracePeriod=30 Jul 07 15:18:57.148080 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:57.148048 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-api-e2e-shared-b-17f250-667fd95f75-hcntc" podUID="ff015d55-c25e-4ae6-adbf-72882996c1a4" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.53:8443/health\": dial tcp 10.133.0.53:8443: connect: connection refused" Jul 07 15:18:57.272896 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:57.272874 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-17f250-667fd95f75-hcntc" Jul 07 15:18:57.411365 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:57.411333 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff015d55-c25e-4ae6-adbf-72882996c1a4-openshift-service-ca\") pod \"ff015d55-c25e-4ae6-adbf-72882996c1a4\" (UID: \"ff015d55-c25e-4ae6-adbf-72882996c1a4\") " Jul 07 15:18:57.411551 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:57.411377 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff015d55-c25e-4ae6-adbf-72882996c1a4-maas-api-tls\") pod \"ff015d55-c25e-4ae6-adbf-72882996c1a4\" (UID: \"ff015d55-c25e-4ae6-adbf-72882996c1a4\") " Jul 07 15:18:57.411551 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:57.411408 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-27pnl\" (UniqueName: \"kubernetes.io/projected/ff015d55-c25e-4ae6-adbf-72882996c1a4-kube-api-access-27pnl\") pod \"ff015d55-c25e-4ae6-adbf-72882996c1a4\" (UID: \"ff015d55-c25e-4ae6-adbf-72882996c1a4\") " Jul 07 15:18:57.411694 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:57.411665 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff015d55-c25e-4ae6-adbf-72882996c1a4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ff015d55-c25e-4ae6-adbf-72882996c1a4" (UID: "ff015d55-c25e-4ae6-adbf-72882996c1a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:18:57.413369 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:57.413349 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff015d55-c25e-4ae6-adbf-72882996c1a4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ff015d55-c25e-4ae6-adbf-72882996c1a4" (UID: "ff015d55-c25e-4ae6-adbf-72882996c1a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:18:57.413469 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:57.413454 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff015d55-c25e-4ae6-adbf-72882996c1a4-kube-api-access-27pnl" (OuterVolumeSpecName: "kube-api-access-27pnl") pod "ff015d55-c25e-4ae6-adbf-72882996c1a4" (UID: "ff015d55-c25e-4ae6-adbf-72882996c1a4"). InnerVolumeSpecName "kube-api-access-27pnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:18:57.512273 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:57.512242 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff015d55-c25e-4ae6-adbf-72882996c1a4-openshift-service-ca\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:18:57.512273 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:57.512272 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff015d55-c25e-4ae6-adbf-72882996c1a4-maas-api-tls\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:18:57.512273 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:57.512282 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27pnl\" (UniqueName: \"kubernetes.io/projected/ff015d55-c25e-4ae6-adbf-72882996c1a4-kube-api-access-27pnl\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:18:57.884765 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:57.884715 2555 generic.go:356] "Generic (PLEG): container finished" podID="ff015d55-c25e-4ae6-adbf-72882996c1a4" containerID="d07a21c8209cf4823e6513b33f973126eac6bcb962be1142aefca2ed2833fb66" exitCode=0 Jul 07 15:18:57.884940 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:57.884782 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-17f250-667fd95f75-hcntc" Jul 07 15:18:57.884940 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:57.884801 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-17f250-667fd95f75-hcntc" event={"ID":"ff015d55-c25e-4ae6-adbf-72882996c1a4","Type":"ContainerDied","Data":"d07a21c8209cf4823e6513b33f973126eac6bcb962be1142aefca2ed2833fb66"} Jul 07 15:18:57.884940 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:57.884854 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-17f250-667fd95f75-hcntc" event={"ID":"ff015d55-c25e-4ae6-adbf-72882996c1a4","Type":"ContainerDied","Data":"39c25c7a4dda361be06522ea424ce871f8bda2960ebf6aa7b33f74897f242998"} Jul 07 15:18:57.884940 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:57.884875 2555 scope.go:117] "RemoveContainer" containerID="d07a21c8209cf4823e6513b33f973126eac6bcb962be1142aefca2ed2833fb66" Jul 07 15:18:57.894659 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:57.894644 2555 scope.go:117] "RemoveContainer" containerID="d07a21c8209cf4823e6513b33f973126eac6bcb962be1142aefca2ed2833fb66" Jul 07 15:18:57.894930 ip-10-0-136-19 kubenswrapper[2555]: E0707 15:18:57.894912 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d07a21c8209cf4823e6513b33f973126eac6bcb962be1142aefca2ed2833fb66\": container with ID starting with d07a21c8209cf4823e6513b33f973126eac6bcb962be1142aefca2ed2833fb66 not found: ID does not exist" containerID="d07a21c8209cf4823e6513b33f973126eac6bcb962be1142aefca2ed2833fb66" Jul 07 15:18:57.894981 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:57.894943 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d07a21c8209cf4823e6513b33f973126eac6bcb962be1142aefca2ed2833fb66"} err="failed to get container status \"d07a21c8209cf4823e6513b33f973126eac6bcb962be1142aefca2ed2833fb66\": rpc error: code = NotFound desc = could not find container \"d07a21c8209cf4823e6513b33f973126eac6bcb962be1142aefca2ed2833fb66\": container with ID starting with d07a21c8209cf4823e6513b33f973126eac6bcb962be1142aefca2ed2833fb66 not found: ID does not exist" Jul 07 15:18:57.908721 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:57.908699 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-17f250-667fd95f75-hcntc"] Jul 07 15:18:57.912773 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:57.912741 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-17f250-667fd95f75-hcntc"] Jul 07 15:18:59.042165 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:18:59.042128 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff015d55-c25e-4ae6-adbf-72882996c1a4" path="/var/lib/kubelet/pods/ff015d55-c25e-4ae6-adbf-72882996c1a4/volumes" Jul 07 15:19:03.455402 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:03.455364 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj"] Jul 07 15:19:03.455889 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:03.455583 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" podUID="4b2d9292-e7b3-431a-8eab-ea585b8ad994" containerName="istio-proxy" containerID="cri-o://153e7014a018837ee4c469e90f3ef04e8ec918f50ae0840ee6c0e6ba786d5614" gracePeriod=30 Jul 07 15:19:08.701359 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:08.701336 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" Jul 07 15:19:08.709806 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:08.709772 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b2d9292-e7b3-431a-8eab-ea585b8ad994-workload-socket\") pod \"4b2d9292-e7b3-431a-8eab-ea585b8ad994\" (UID: \"4b2d9292-e7b3-431a-8eab-ea585b8ad994\") " Jul 07 15:19:08.709958 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:08.709832 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b2d9292-e7b3-431a-8eab-ea585b8ad994-istio-podinfo\") pod \"4b2d9292-e7b3-431a-8eab-ea585b8ad994\" (UID: \"4b2d9292-e7b3-431a-8eab-ea585b8ad994\") " Jul 07 15:19:08.709958 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:08.709854 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b2d9292-e7b3-431a-8eab-ea585b8ad994-credential-socket\") pod \"4b2d9292-e7b3-431a-8eab-ea585b8ad994\" (UID: \"4b2d9292-e7b3-431a-8eab-ea585b8ad994\") " Jul 07 15:19:08.709958 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:08.709887 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b2d9292-e7b3-431a-8eab-ea585b8ad994-istio-token\") pod \"4b2d9292-e7b3-431a-8eab-ea585b8ad994\" (UID: \"4b2d9292-e7b3-431a-8eab-ea585b8ad994\") " Jul 07 15:19:08.709958 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:08.709911 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b2d9292-e7b3-431a-8eab-ea585b8ad994-istio-envoy\") pod \"4b2d9292-e7b3-431a-8eab-ea585b8ad994\" (UID: \"4b2d9292-e7b3-431a-8eab-ea585b8ad994\") " Jul 07 15:19:08.710160 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:08.709964 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b2d9292-e7b3-431a-8eab-ea585b8ad994-istio-data\") pod \"4b2d9292-e7b3-431a-8eab-ea585b8ad994\" (UID: \"4b2d9292-e7b3-431a-8eab-ea585b8ad994\") " Jul 07 15:19:08.710160 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:08.710003 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b2d9292-e7b3-431a-8eab-ea585b8ad994-workload-certs\") pod \"4b2d9292-e7b3-431a-8eab-ea585b8ad994\" (UID: \"4b2d9292-e7b3-431a-8eab-ea585b8ad994\") " Jul 07 15:19:08.710160 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:08.710012 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b2d9292-e7b3-431a-8eab-ea585b8ad994-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4b2d9292-e7b3-431a-8eab-ea585b8ad994" (UID: "4b2d9292-e7b3-431a-8eab-ea585b8ad994"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:19:08.710160 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:08.710066 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b2d9292-e7b3-431a-8eab-ea585b8ad994-istiod-ca-cert\") pod \"4b2d9292-e7b3-431a-8eab-ea585b8ad994\" (UID: \"4b2d9292-e7b3-431a-8eab-ea585b8ad994\") " Jul 07 15:19:08.710160 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:08.710113 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8z8j8\" (UniqueName: \"kubernetes.io/projected/4b2d9292-e7b3-431a-8eab-ea585b8ad994-kube-api-access-8z8j8\") pod \"4b2d9292-e7b3-431a-8eab-ea585b8ad994\" (UID: \"4b2d9292-e7b3-431a-8eab-ea585b8ad994\") " Jul 07 15:19:08.710426 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:08.710154 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b2d9292-e7b3-431a-8eab-ea585b8ad994-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4b2d9292-e7b3-431a-8eab-ea585b8ad994" (UID: "4b2d9292-e7b3-431a-8eab-ea585b8ad994"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:19:08.710426 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:08.710232 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b2d9292-e7b3-431a-8eab-ea585b8ad994-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4b2d9292-e7b3-431a-8eab-ea585b8ad994" (UID: "4b2d9292-e7b3-431a-8eab-ea585b8ad994"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:19:08.710426 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:08.710406 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b2d9292-e7b3-431a-8eab-ea585b8ad994-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4b2d9292-e7b3-431a-8eab-ea585b8ad994" (UID: "4b2d9292-e7b3-431a-8eab-ea585b8ad994"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:19:08.710585 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:08.710483 2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b2d9292-e7b3-431a-8eab-ea585b8ad994-credential-socket\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:19:08.710585 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:08.710501 2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b2d9292-e7b3-431a-8eab-ea585b8ad994-workload-certs\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:19:08.710585 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:08.710515 2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b2d9292-e7b3-431a-8eab-ea585b8ad994-istiod-ca-cert\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:19:08.710585 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:08.710527 2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b2d9292-e7b3-431a-8eab-ea585b8ad994-workload-socket\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:19:08.711129 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:08.711089 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b2d9292-e7b3-431a-8eab-ea585b8ad994-istio-data" (OuterVolumeSpecName: "istio-data") pod "4b2d9292-e7b3-431a-8eab-ea585b8ad994" (UID: "4b2d9292-e7b3-431a-8eab-ea585b8ad994"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:19:08.715244 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:08.712873 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b2d9292-e7b3-431a-8eab-ea585b8ad994-kube-api-access-8z8j8" (OuterVolumeSpecName: "kube-api-access-8z8j8") pod "4b2d9292-e7b3-431a-8eab-ea585b8ad994" (UID: "4b2d9292-e7b3-431a-8eab-ea585b8ad994"). InnerVolumeSpecName "kube-api-access-8z8j8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:19:08.718352 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:08.718273 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b2d9292-e7b3-431a-8eab-ea585b8ad994-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4b2d9292-e7b3-431a-8eab-ea585b8ad994" (UID: "4b2d9292-e7b3-431a-8eab-ea585b8ad994"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 15:19:08.718875 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:08.718849 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b2d9292-e7b3-431a-8eab-ea585b8ad994-istio-token" (OuterVolumeSpecName: "istio-token") pod "4b2d9292-e7b3-431a-8eab-ea585b8ad994" (UID: "4b2d9292-e7b3-431a-8eab-ea585b8ad994"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:19:08.719040 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:08.719018 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b2d9292-e7b3-431a-8eab-ea585b8ad994-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4b2d9292-e7b3-431a-8eab-ea585b8ad994" (UID: "4b2d9292-e7b3-431a-8eab-ea585b8ad994"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:19:08.811131 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:08.811105 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z8j8\" (UniqueName: \"kubernetes.io/projected/4b2d9292-e7b3-431a-8eab-ea585b8ad994-kube-api-access-8z8j8\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:19:08.811131 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:08.811132 2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b2d9292-e7b3-431a-8eab-ea585b8ad994-istio-podinfo\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:19:08.811299 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:08.811141 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b2d9292-e7b3-431a-8eab-ea585b8ad994-istio-token\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:19:08.811299 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:08.811149 2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b2d9292-e7b3-431a-8eab-ea585b8ad994-istio-envoy\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:19:08.811299 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:08.811158 2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b2d9292-e7b3-431a-8eab-ea585b8ad994-istio-data\") on node \"ip-10-0-136-19.ec2.internal\" DevicePath \"\"" Jul 07 15:19:08.938142 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:08.938108 2555 generic.go:356] "Generic (PLEG): container finished" podID="4b2d9292-e7b3-431a-8eab-ea585b8ad994" containerID="153e7014a018837ee4c469e90f3ef04e8ec918f50ae0840ee6c0e6ba786d5614" exitCode=0 Jul 07 15:19:08.938306 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:08.938174 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" Jul 07 15:19:08.938306 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:08.938187 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" event={"ID":"4b2d9292-e7b3-431a-8eab-ea585b8ad994","Type":"ContainerDied","Data":"153e7014a018837ee4c469e90f3ef04e8ec918f50ae0840ee6c0e6ba786d5614"} Jul 07 15:19:08.938306 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:08.938276 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj" event={"ID":"4b2d9292-e7b3-431a-8eab-ea585b8ad994","Type":"ContainerDied","Data":"02fae47fab5e181435aefe8a900d693c2250b1951ed2004cd0b6a8f5f96db2f0"} Jul 07 15:19:08.938306 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:08.938299 2555 scope.go:117] "RemoveContainer" containerID="153e7014a018837ee4c469e90f3ef04e8ec918f50ae0840ee6c0e6ba786d5614" Jul 07 15:19:08.947286 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:08.947271 2555 scope.go:117] "RemoveContainer" containerID="0c70972de439cc7ef006381c2719a0756a88c22ce0978f2f50a4749c349218fa" Jul 07 15:19:08.955015 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:08.955000 2555 scope.go:117] "RemoveContainer" containerID="153e7014a018837ee4c469e90f3ef04e8ec918f50ae0840ee6c0e6ba786d5614" Jul 07 15:19:08.955249 ip-10-0-136-19 kubenswrapper[2555]: E0707 15:19:08.955228 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"153e7014a018837ee4c469e90f3ef04e8ec918f50ae0840ee6c0e6ba786d5614\": container with ID starting with 153e7014a018837ee4c469e90f3ef04e8ec918f50ae0840ee6c0e6ba786d5614 not found: ID does not exist" containerID="153e7014a018837ee4c469e90f3ef04e8ec918f50ae0840ee6c0e6ba786d5614" Jul 07 15:19:08.955332 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:08.955260 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"153e7014a018837ee4c469e90f3ef04e8ec918f50ae0840ee6c0e6ba786d5614"} err="failed to get container status \"153e7014a018837ee4c469e90f3ef04e8ec918f50ae0840ee6c0e6ba786d5614\": rpc error: code = NotFound desc = could not find container \"153e7014a018837ee4c469e90f3ef04e8ec918f50ae0840ee6c0e6ba786d5614\": container with ID starting with 153e7014a018837ee4c469e90f3ef04e8ec918f50ae0840ee6c0e6ba786d5614 not found: ID does not exist" Jul 07 15:19:08.955332 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:08.955283 2555 scope.go:117] "RemoveContainer" containerID="0c70972de439cc7ef006381c2719a0756a88c22ce0978f2f50a4749c349218fa" Jul 07 15:19:08.955511 ip-10-0-136-19 kubenswrapper[2555]: E0707 15:19:08.955493 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c70972de439cc7ef006381c2719a0756a88c22ce0978f2f50a4749c349218fa\": container with ID starting with 0c70972de439cc7ef006381c2719a0756a88c22ce0978f2f50a4749c349218fa not found: ID does not exist" containerID="0c70972de439cc7ef006381c2719a0756a88c22ce0978f2f50a4749c349218fa" Jul 07 15:19:08.955554 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:08.955517 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c70972de439cc7ef006381c2719a0756a88c22ce0978f2f50a4749c349218fa"} err="failed to get container status \"0c70972de439cc7ef006381c2719a0756a88c22ce0978f2f50a4749c349218fa\": rpc error: code = NotFound desc = could not find container \"0c70972de439cc7ef006381c2719a0756a88c22ce0978f2f50a4749c349218fa\": container with ID starting with 0c70972de439cc7ef006381c2719a0756a88c22ce0978f2f50a4749c349218fa not found: ID does not exist" Jul 07 15:19:08.962648 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:08.962625 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj"] Jul 07 15:19:08.966529 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:08.966506 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-17f250-openshift-default-77f6f7bc-zkmbj"] Jul 07 15:19:08.986145 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:08.986100 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-767c6f79c5-drmrr_776b8309-f958-440a-b03f-3f054c8b8e68/maas-api/0.log" Jul 07 15:19:09.041523 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:09.041485 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b2d9292-e7b3-431a-8eab-ea585b8ad994" path="/var/lib/kubelet/pods/4b2d9292-e7b3-431a-8eab-ea585b8ad994/volumes" Jul 07 15:19:09.341455 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:09.341428 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5b667b448b-k5gc4_bb97ab4b-9777-44dc-9ba6-646e64b881ea/manager/0.log" Jul 07 15:19:09.828165 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:09.828141 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-655bb48ff6-q7dpz_2620af16-036c-4f11-8e0f-3392581bf237/copy-manifests/0.log" Jul 07 15:19:09.835503 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:09.835480 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-655bb48ff6-q7dpz_2620af16-036c-4f11-8e0f-3392581bf237/manager/0.log" Jul 07 15:19:12.641937 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:12.641873 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678vncl5_645fefe7-23f5-414c-b03e-bbba6e8ce6ca/istio-proxy/0.log" Jul 07 15:19:12.767802 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:12.767774 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-p7bmz_84e8a09b-f81e-409b-87cd-b754240f58f8/discovery/0.log" Jul 07 15:19:13.485864 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:13.485832 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8578b59545-z44lc_76dd393c-d59d-4978-83f6-7bcdab3169c8/router/0.log" Jul 07 15:19:13.839286 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:13.839264 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-s65rw_b682a975-ce9b-488a-a3b8-5d33d73cb1c1/main/0.log" Jul 07 15:19:14.293083 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:14.293007 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-zwfl5_0c22f9aa-b3b7-4db2-babb-0f9924d3a2c2/main/0.log" Jul 07 15:19:21.639094 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:21.639064 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bq6hn_686ecb3c-7c9a-4ac1-ac27-4430651b0870/global-pull-secret-syncer/0.log" Jul 07 15:19:21.802766 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:21.802702 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4zclx_3ea25b41-29d9-4e81-b4d5-6252d76a8e66/konnectivity-agent/0.log" Jul 07 15:19:21.881327 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:21.881297 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-19.ec2.internal_ff9a5e8afa26523332eda2af110e42ee/haproxy/0.log" Jul 07 15:19:29.089234 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:29.089201 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-4zhnv_a237e875-6fcb-4b05-bfff-6742c35b7afa/cluster-monitoring-operator/0.log" Jul 07 15:19:29.119014 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:29.118989 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-jbflt_afa77f75-9627-4138-9d5c-83e060022fdf/kube-state-metrics/0.log" Jul 07 15:19:29.144892 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:29.144819 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-jbflt_afa77f75-9627-4138-9d5c-83e060022fdf/kube-rbac-proxy-main/0.log" Jul 07 15:19:29.169421 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:29.169386 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-jbflt_afa77f75-9627-4138-9d5c-83e060022fdf/kube-rbac-proxy-self/0.log" Jul 07 15:19:29.438114 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:29.438027 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xhlln_6b68f320-00d3-41cb-a818-39d5f50c6d07/node-exporter/0.log" Jul 07 15:19:29.473309 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:29.473279 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xhlln_6b68f320-00d3-41cb-a818-39d5f50c6d07/kube-rbac-proxy/0.log" Jul 07 15:19:29.496043 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:29.496015 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xhlln_6b68f320-00d3-41cb-a818-39d5f50c6d07/init-textfile/0.log" Jul 07 15:19:29.606240 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:29.606213 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d6cf9f09-a0c7-45a6-9330-a10ae98c2906/prometheus/0.log" Jul 07 15:19:29.630635 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:29.630613 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d6cf9f09-a0c7-45a6-9330-a10ae98c2906/config-reloader/0.log" Jul 07 15:19:29.676601 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:29.676571 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d6cf9f09-a0c7-45a6-9330-a10ae98c2906/thanos-sidecar/0.log" Jul 07 15:19:29.701269 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:29.701201 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d6cf9f09-a0c7-45a6-9330-a10ae98c2906/kube-rbac-proxy-web/0.log" Jul 07 15:19:29.727034 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:29.726994 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d6cf9f09-a0c7-45a6-9330-a10ae98c2906/kube-rbac-proxy/0.log" Jul 07 15:19:29.749498 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:29.749459 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d6cf9f09-a0c7-45a6-9330-a10ae98c2906/kube-rbac-proxy-thanos/0.log" Jul 07 15:19:29.773023 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:29.773006 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d6cf9f09-a0c7-45a6-9330-a10ae98c2906/init-config-reloader/0.log" Jul 07 15:19:31.409319 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:31.409282 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-psm49/perf-node-gather-daemonset-5z95m"] Jul 07 15:19:31.409709 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:31.409607 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b2d9292-e7b3-431a-8eab-ea585b8ad994" containerName="istio-proxy" Jul 07 15:19:31.409709 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:31.409617 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b2d9292-e7b3-431a-8eab-ea585b8ad994" containerName="istio-proxy" Jul 07 15:19:31.409709 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:31.409625 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff015d55-c25e-4ae6-adbf-72882996c1a4" containerName="maas-api" Jul 07 15:19:31.409709 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:31.409630 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff015d55-c25e-4ae6-adbf-72882996c1a4" containerName="maas-api" Jul 07 15:19:31.409709 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:31.409651 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cd20114-e2cc-42cc-adc7-28c860b2a629" containerName="maas-api" Jul 07 15:19:31.409709 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:31.409656 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cd20114-e2cc-42cc-adc7-28c860b2a629" containerName="maas-api" Jul 07 15:19:31.409709 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:31.409664 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b2d9292-e7b3-431a-8eab-ea585b8ad994" containerName="istio-proxy" Jul 07 15:19:31.409709 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:31.409669 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b2d9292-e7b3-431a-8eab-ea585b8ad994" containerName="istio-proxy" Jul 07 15:19:31.410137 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:31.409737 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff015d55-c25e-4ae6-adbf-72882996c1a4" containerName="maas-api" Jul 07 15:19:31.410137 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:31.409795 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cd20114-e2cc-42cc-adc7-28c860b2a629" containerName="maas-api" Jul 07 15:19:31.410137 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:31.409808 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b2d9292-e7b3-431a-8eab-ea585b8ad994" containerName="istio-proxy" Jul 07 15:19:31.410137 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:31.409815 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b2d9292-e7b3-431a-8eab-ea585b8ad994" containerName="istio-proxy" Jul 07 15:19:31.413115 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:31.413095 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-psm49/perf-node-gather-daemonset-5z95m" Jul 07 15:19:31.415872 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:31.415845 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-psm49\"/\"default-dockercfg-rg4cq\"" Jul 07 15:19:31.416014 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:31.415957 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-psm49\"/\"kube-root-ca.crt\"" Jul 07 15:19:31.416060 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:31.416041 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-psm49\"/\"openshift-service-ca.crt\"" Jul 07 15:19:31.423606 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:31.423586 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-psm49/perf-node-gather-daemonset-5z95m"] Jul 07 15:19:31.507140 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:31.507111 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5qj7\" (UniqueName: \"kubernetes.io/projected/42824883-48b6-4baf-ac81-5087461e2c9b-kube-api-access-l5qj7\") pod \"perf-node-gather-daemonset-5z95m\" (UID: \"42824883-48b6-4baf-ac81-5087461e2c9b\") " pod="openshift-must-gather-psm49/perf-node-gather-daemonset-5z95m" Jul 07 15:19:31.507295 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:31.507146 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/42824883-48b6-4baf-ac81-5087461e2c9b-podres\") pod \"perf-node-gather-daemonset-5z95m\" (UID: \"42824883-48b6-4baf-ac81-5087461e2c9b\") " pod="openshift-must-gather-psm49/perf-node-gather-daemonset-5z95m" Jul 07 15:19:31.507295 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:31.507168 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42824883-48b6-4baf-ac81-5087461e2c9b-sys\") pod \"perf-node-gather-daemonset-5z95m\" (UID: \"42824883-48b6-4baf-ac81-5087461e2c9b\") " pod="openshift-must-gather-psm49/perf-node-gather-daemonset-5z95m" Jul 07 15:19:31.507295 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:31.507207 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/42824883-48b6-4baf-ac81-5087461e2c9b-lib-modules\") pod \"perf-node-gather-daemonset-5z95m\" (UID: \"42824883-48b6-4baf-ac81-5087461e2c9b\") " pod="openshift-must-gather-psm49/perf-node-gather-daemonset-5z95m" Jul 07 15:19:31.507295 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:31.507281 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/42824883-48b6-4baf-ac81-5087461e2c9b-proc\") pod \"perf-node-gather-daemonset-5z95m\" (UID: \"42824883-48b6-4baf-ac81-5087461e2c9b\") " pod="openshift-must-gather-psm49/perf-node-gather-daemonset-5z95m" Jul 07 15:19:31.608008 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:31.607980 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/42824883-48b6-4baf-ac81-5087461e2c9b-proc\") pod \"perf-node-gather-daemonset-5z95m\" (UID: \"42824883-48b6-4baf-ac81-5087461e2c9b\") " pod="openshift-must-gather-psm49/perf-node-gather-daemonset-5z95m" Jul 07 15:19:31.608172 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:31.608059 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5qj7\" (UniqueName: \"kubernetes.io/projected/42824883-48b6-4baf-ac81-5087461e2c9b-kube-api-access-l5qj7\") pod \"perf-node-gather-daemonset-5z95m\" (UID: \"42824883-48b6-4baf-ac81-5087461e2c9b\") " pod="openshift-must-gather-psm49/perf-node-gather-daemonset-5z95m" Jul 07 15:19:31.608172 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:31.608084 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/42824883-48b6-4baf-ac81-5087461e2c9b-podres\") pod \"perf-node-gather-daemonset-5z95m\" (UID: \"42824883-48b6-4baf-ac81-5087461e2c9b\") " pod="openshift-must-gather-psm49/perf-node-gather-daemonset-5z95m" Jul 07 15:19:31.608172 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:31.608102 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42824883-48b6-4baf-ac81-5087461e2c9b-sys\") pod \"perf-node-gather-daemonset-5z95m\" (UID: \"42824883-48b6-4baf-ac81-5087461e2c9b\") " pod="openshift-must-gather-psm49/perf-node-gather-daemonset-5z95m" Jul 07 15:19:31.608172 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:31.608107 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/42824883-48b6-4baf-ac81-5087461e2c9b-proc\") pod \"perf-node-gather-daemonset-5z95m\" (UID: \"42824883-48b6-4baf-ac81-5087461e2c9b\") " pod="openshift-must-gather-psm49/perf-node-gather-daemonset-5z95m" Jul 07 15:19:31.608172 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:31.608124 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/42824883-48b6-4baf-ac81-5087461e2c9b-lib-modules\") pod \"perf-node-gather-daemonset-5z95m\" (UID: \"42824883-48b6-4baf-ac81-5087461e2c9b\") " pod="openshift-must-gather-psm49/perf-node-gather-daemonset-5z95m" Jul 07 15:19:31.608335 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:31.608190 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42824883-48b6-4baf-ac81-5087461e2c9b-sys\") pod \"perf-node-gather-daemonset-5z95m\" (UID: \"42824883-48b6-4baf-ac81-5087461e2c9b\") " pod="openshift-must-gather-psm49/perf-node-gather-daemonset-5z95m" Jul 07 15:19:31.608335 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:31.608242 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/42824883-48b6-4baf-ac81-5087461e2c9b-podres\") pod \"perf-node-gather-daemonset-5z95m\" (UID: \"42824883-48b6-4baf-ac81-5087461e2c9b\") " pod="openshift-must-gather-psm49/perf-node-gather-daemonset-5z95m" Jul 07 15:19:31.608335 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:31.608245 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/42824883-48b6-4baf-ac81-5087461e2c9b-lib-modules\") pod \"perf-node-gather-daemonset-5z95m\" (UID: \"42824883-48b6-4baf-ac81-5087461e2c9b\") " pod="openshift-must-gather-psm49/perf-node-gather-daemonset-5z95m" Jul 07 15:19:31.616617 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:31.616591 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5qj7\" (UniqueName: \"kubernetes.io/projected/42824883-48b6-4baf-ac81-5087461e2c9b-kube-api-access-l5qj7\") pod \"perf-node-gather-daemonset-5z95m\" (UID: \"42824883-48b6-4baf-ac81-5087461e2c9b\") " pod="openshift-must-gather-psm49/perf-node-gather-daemonset-5z95m" Jul 07 15:19:31.723530 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:31.723460 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-psm49/perf-node-gather-daemonset-5z95m" Jul 07 15:19:31.854363 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:31.854338 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-psm49/perf-node-gather-daemonset-5z95m"] Jul 07 15:19:31.858362 ip-10-0-136-19 kubenswrapper[2555]: W0707 15:19:31.858327 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod42824883_48b6_4baf_ac81_5087461e2c9b.slice/crio-96aa3f58ab43ceef91116f398b0aeb508d354f3e3a850803b71970a0bf759dda WatchSource:0}: Error finding container 96aa3f58ab43ceef91116f398b0aeb508d354f3e3a850803b71970a0bf759dda: Status 404 returned error can't find the container with id 96aa3f58ab43ceef91116f398b0aeb508d354f3e3a850803b71970a0bf759dda Jul 07 15:19:32.041610 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:32.041528 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-psm49/perf-node-gather-daemonset-5z95m" event={"ID":"42824883-48b6-4baf-ac81-5087461e2c9b","Type":"ContainerStarted","Data":"03b037b7c641df23fa8e3018aca14065c0d80218316aa2cc3debf85a2906c75f"} Jul 07 15:19:32.041610 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:32.041572 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-psm49/perf-node-gather-daemonset-5z95m" event={"ID":"42824883-48b6-4baf-ac81-5087461e2c9b","Type":"ContainerStarted","Data":"96aa3f58ab43ceef91116f398b0aeb508d354f3e3a850803b71970a0bf759dda"} Jul 07 15:19:32.041610 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:32.041604 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-psm49/perf-node-gather-daemonset-5z95m" Jul 07 15:19:32.061132 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:32.061087 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-psm49/perf-node-gather-daemonset-5z95m" podStartSLOduration=1.061072042 podStartE2EDuration="1.061072042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:19:32.059280941 +0000 UTC m=+2713.483467415" watchObservedRunningTime="2026-07-07 15:19:32.061072042 +0000 UTC m=+2713.485258514" Jul 07 15:19:32.870957 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:32.870910 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-4jw8p_a6580919-d925-4965-ba84-e035fa29ad2a/volume-data-source-validator/0.log" Jul 07 15:19:33.876230 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:33.876192 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9pd7h_c447425e-fb20-4ac8-b7e7-ed07d60953e5/dns/0.log" Jul 07 15:19:33.899716 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:33.899685 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9pd7h_c447425e-fb20-4ac8-b7e7-ed07d60953e5/kube-rbac-proxy/0.log" Jul 07 15:19:34.024543 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:34.024515 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-czppd_553f749a-649c-41c0-aa39-a0cb13725ca8/dns-node-resolver/0.log" Jul 07 15:19:34.671959 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:34.671896 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vkwws_7d94747d-5e8a-4eab-8234-969f7dfbaee5/node-ca/0.log" Jul 07 15:19:35.679185 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:35.679157 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678vncl5_645fefe7-23f5-414c-b03e-bbba6e8ce6ca/istio-proxy/0.log" Jul 07 15:19:35.793042 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:35.793009 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-p7bmz_84e8a09b-f81e-409b-87cd-b754240f58f8/discovery/0.log" Jul 07 15:19:36.020182 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:36.020100 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8578b59545-z44lc_76dd393c-d59d-4978-83f6-7bcdab3169c8/router/0.log" Jul 07 15:19:36.742926 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:36.742896 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xv4vf_8c25e009-ed68-418d-b5c1-02d8fa948f74/serve-healthcheck-canary/0.log" Jul 07 15:19:37.399039 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:37.399009 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mpm2g_73b85bdb-2070-4cfa-a85c-82f2f2065d6c/kube-rbac-proxy/0.log" Jul 07 15:19:37.428670 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:37.428649 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mpm2g_73b85bdb-2070-4cfa-a85c-82f2f2065d6c/exporter/0.log" Jul 07 15:19:37.453531 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:37.453512 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mpm2g_73b85bdb-2070-4cfa-a85c-82f2f2065d6c/extractor/0.log" Jul 07 15:19:38.055741 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:38.055716 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-psm49/perf-node-gather-daemonset-5z95m" Jul 07 15:19:39.664357 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:39.664322 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-767c6f79c5-drmrr_776b8309-f958-440a-b03f-3f054c8b8e68/maas-api/0.log" Jul 07 15:19:39.824626 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:39.824577 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5b667b448b-k5gc4_bb97ab4b-9777-44dc-9ba6-646e64b881ea/manager/0.log" Jul 07 15:19:40.071955 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:40.071857 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-655bb48ff6-q7dpz_2620af16-036c-4f11-8e0f-3392581bf237/manager/0.log" Jul 07 15:19:40.096822 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:40.096789 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-655bb48ff6-q7dpz_2620af16-036c-4f11-8e0f-3392581bf237/copy-manifests/0.log" Jul 07 15:19:41.078627 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:41.078601 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678vncl5_645fefe7-23f5-414c-b03e-bbba6e8ce6ca/istio-proxy/0.log" Jul 07 15:19:41.177448 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:41.177418 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-p7bmz_84e8a09b-f81e-409b-87cd-b754240f58f8/discovery/0.log" Jul 07 15:19:41.345410 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:41.345336 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8578b59545-z44lc_76dd393c-d59d-4978-83f6-7bcdab3169c8/router/0.log" Jul 07 15:19:41.611877 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:41.611811 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6d85f8474c-8mn9v_ea222582-1fd6-4e76-a6cb-18a353b998c3/manager/0.log" Jul 07 15:19:41.669896 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:41.669854 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-8nzr4_9a2430f4-bf79-448a-8e67-0e347e014344/openshift-lws-operator/0.log" Jul 07 15:19:42.685198 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:42.685173 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678vncl5_645fefe7-23f5-414c-b03e-bbba6e8ce6ca/istio-proxy/0.log" Jul 07 15:19:42.775640 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:42.775610 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-p7bmz_84e8a09b-f81e-409b-87cd-b754240f58f8/discovery/0.log" Jul 07 15:19:42.928582 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:42.928501 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8578b59545-z44lc_76dd393c-d59d-4978-83f6-7bcdab3169c8/router/0.log" Jul 07 15:19:46.948115 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:46.948076 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-b56rb_75aeac0c-a9eb-44a9-9ed1-d7412dab8785/kube-storage-version-migrator-operator/1.log" Jul 07 15:19:46.949077 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:46.949058 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-b56rb_75aeac0c-a9eb-44a9-9ed1-d7412dab8785/kube-storage-version-migrator-operator/0.log" Jul 07 15:19:48.048113 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:48.048075 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9t7xp_89159896-e9f7-42e0-93e7-23122a296d8c/kube-multus-additional-cni-plugins/0.log" Jul 07 15:19:48.069975 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:48.069951 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9t7xp_89159896-e9f7-42e0-93e7-23122a296d8c/egress-router-binary-copy/0.log" Jul 07 15:19:48.091359 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:48.091339 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9t7xp_89159896-e9f7-42e0-93e7-23122a296d8c/cni-plugins/0.log" Jul 07 15:19:48.112945 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:48.112922 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9t7xp_89159896-e9f7-42e0-93e7-23122a296d8c/bond-cni-plugin/0.log" Jul 07 15:19:48.133588 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:48.133568 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9t7xp_89159896-e9f7-42e0-93e7-23122a296d8c/routeoverride-cni/0.log" Jul 07 15:19:48.154497 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:48.154476 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9t7xp_89159896-e9f7-42e0-93e7-23122a296d8c/whereabouts-cni-bincopy/0.log" Jul 07 15:19:48.175573 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:48.175552 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9t7xp_89159896-e9f7-42e0-93e7-23122a296d8c/whereabouts-cni/0.log" Jul 07 15:19:48.370534 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:48.370505 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-bfcr2_f5f5ff03-a3bc-4758-8023-e023ab4ed3b3/kube-multus/0.log" Jul 07 15:19:48.492335 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:48.492314 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lpmtl_e3bbd20c-73f5-4d24-aed9-145e4b89a590/network-metrics-daemon/0.log" Jul 07 15:19:48.512460 ip-10-0-136-19 kubenswrapper[2555]: I0707 15:19:48.512439 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lpmtl_e3bbd20c-73f5-4d24-aed9-145e4b89a590/kube-rbac-proxy/0.log"