Jul 06 19:26:29.215920 ip-10-0-135-23 systemd[1]: Starting Kubernetes Kubelet... Jul 06 19:26:31.570556 ip-10-0-135-23 systemd[1]: Started Kubernetes Kubelet. Jul 06 19:26:31.589372 ip-10-0-135-23 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 19:26:31.589372 ip-10-0-135-23 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 19:26:31.589372 ip-10-0-135-23 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 19:26:31.589372 ip-10-0-135-23 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 19:26:31.589372 ip-10-0-135-23 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 19:26:31.589372 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.495662 2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 19:26:31.589372 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505406 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:26:31.589372 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505534 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:26:31.589372 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505541 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:26:31.589372 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505545 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:26:31.589372 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505548 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:26:31.589372 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505551 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:26:31.589372 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505554 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:26:31.589372 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505558 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:26:31.591472 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505561 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:26:31.591472 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505563 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:26:31.591472 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505566 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:26:31.591472 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505569 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:26:31.591472 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505572 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:26:31.591472 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505574 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:26:31.591472 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505577 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:26:31.591472 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505579 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:26:31.591472 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505582 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:26:31.591472 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505584 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:26:31.591472 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505587 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:26:31.591472 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505591 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:26:31.591472 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505596 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:26:31.591472 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505600 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:26:31.591472 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505603 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:26:31.591472 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505605 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:26:31.591472 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505609 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:26:31.591472 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505612 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:26:31.591472 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505615 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:26:31.591472 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505618 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:26:31.592583 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505621 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:26:31.592583 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505623 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:26:31.592583 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505626 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:26:31.592583 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505628 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:26:31.592583 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505631 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:26:31.592583 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505634 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:26:31.592583 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505636 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:26:31.592583 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505639 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:26:31.592583 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505641 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:26:31.592583 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505644 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:26:31.592583 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505648 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:26:31.592583 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505651 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:26:31.592583 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505653 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:26:31.592583 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505656 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:26:31.592583 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505658 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:26:31.592583 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505661 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:26:31.592583 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505663 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:26:31.592583 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505666 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:26:31.592583 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505668 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:26:31.592583 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505671 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:26:31.594083 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505675 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:26:31.594083 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505679 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:26:31.594083 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505681 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:26:31.594083 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505684 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:26:31.594083 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505687 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:26:31.594083 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505690 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:26:31.594083 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505692 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:26:31.594083 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505695 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:26:31.594083 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505698 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:26:31.594083 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505701 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:26:31.594083 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505703 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:26:31.594083 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505706 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:26:31.594083 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505709 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:26:31.594083 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505712 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:26:31.594083 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505714 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:26:31.594083 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505717 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:26:31.594083 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505719 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:26:31.594083 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505722 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:26:31.594083 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505724 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:26:31.594768 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505727 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:26:31.594768 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505730 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:26:31.594768 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505732 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:26:31.594768 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505735 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:26:31.594768 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505737 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:26:31.594768 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505739 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:26:31.594768 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505742 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:26:31.594768 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505744 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:26:31.594768 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505747 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:26:31.594768 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505749 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:26:31.594768 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505751 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:26:31.594768 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505754 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:26:31.594768 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505757 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:26:31.594768 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505761 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:26:31.594768 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505764 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:26:31.594768 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505767 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:26:31.594768 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505769 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:26:31.594768 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505773 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:26:31.594768 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505775 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:26:31.594768 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505778 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:26:31.595443 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505780 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:26:31.595443 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505783 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:26:31.595443 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505786 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:26:31.595443 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505790 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:26:31.595443 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505793 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:26:31.595443 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505796 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:26:31.595443 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505799 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:26:31.595443 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505802 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:26:31.595443 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505805 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:26:31.595443 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505807 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:26:31.595443 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505810 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:26:31.595443 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.505813 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:26:31.595443 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506541 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:26:31.595443 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506549 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:26:31.595443 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506553 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:26:31.595443 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506556 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:26:31.595443 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506560 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:26:31.595443 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506562 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:26:31.595443 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506565 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:26:31.595443 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506568 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:26:31.596071 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506571 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:26:31.596071 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506573 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:26:31.596071 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506576 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:26:31.596071 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506579 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:26:31.596071 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506581 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:26:31.596071 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506584 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:26:31.596071 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506586 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:26:31.596071 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506589 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:26:31.596071 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506591 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:26:31.596071 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506595 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:26:31.596071 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506598 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:26:31.596071 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506600 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:26:31.596071 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506603 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:26:31.596071 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506605 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:26:31.596071 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506608 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:26:31.596071 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506611 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:26:31.596071 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506613 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:26:31.596071 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506616 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:26:31.596071 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506618 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:26:31.596764 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506622 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:26:31.596764 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506626 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:26:31.596764 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506629 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:26:31.596764 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506632 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:26:31.596764 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506636 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:26:31.596764 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506640 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:26:31.596764 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506643 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:26:31.596764 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506647 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:26:31.596764 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506650 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:26:31.596764 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506653 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:26:31.596764 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506656 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:26:31.596764 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506659 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:26:31.596764 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506662 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:26:31.596764 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506665 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:26:31.596764 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506668 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:26:31.596764 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506671 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:26:31.596764 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506673 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:26:31.596764 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506676 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:26:31.596764 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506679 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:26:31.597440 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506681 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:26:31.597440 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506684 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:26:31.597440 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506687 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:26:31.597440 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506689 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:26:31.597440 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506692 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:26:31.597440 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506694 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:26:31.597440 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506697 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:26:31.597440 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506699 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:26:31.597440 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506702 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:26:31.597440 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506705 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:26:31.597440 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506707 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:26:31.597440 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506709 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:26:31.597440 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506713 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:26:31.597440 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506715 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:26:31.597440 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506719 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:26:31.597440 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506723 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:26:31.597440 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506726 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:26:31.597440 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506729 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:26:31.597440 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506732 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:26:31.598072 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506734 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:26:31.598072 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506737 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:26:31.598072 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506740 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:26:31.598072 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506743 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:26:31.598072 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506745 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:26:31.598072 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506748 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:26:31.598072 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506750 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:26:31.598072 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506753 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:26:31.598072 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506755 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:26:31.598072 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506758 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:26:31.598072 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506760 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:26:31.598072 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506763 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:26:31.598072 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506766 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:26:31.598072 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506769 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:26:31.598072 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506771 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:26:31.598072 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506774 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:26:31.598072 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506776 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:26:31.598072 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506779 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:26:31.598072 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506782 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:26:31.598072 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506784 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:26:31.598072 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506787 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:26:31.598858 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506789 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:26:31.598858 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506791 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:26:31.598858 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506794 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:26:31.598858 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506797 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:26:31.598858 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506800 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:26:31.598858 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506802 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:26:31.598858 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506805 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:26:31.598858 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506807 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:26:31.598858 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506810 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:26:31.598858 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506812 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:26:31.598858 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506815 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:26:31.598858 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506817 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:26:31.598858 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.506820 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:26:31.598858 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507251 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:26:31.598858 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507257 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:26:31.598858 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507261 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:26:31.598858 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507264 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:26:31.598858 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507267 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:26:31.598858 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507269 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:26:31.599474 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507273 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:26:31.599474 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507275 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:26:31.599474 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507278 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:26:31.599474 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507281 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:26:31.599474 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507284 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:26:31.599474 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507287 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:26:31.599474 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507289 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:26:31.599474 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507292 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:26:31.599474 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507295 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:26:31.599474 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507297 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:26:31.599474 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507301 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:26:31.599474 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507303 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:26:31.599474 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507306 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:26:31.599474 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507308 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:26:31.599474 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507311 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:26:31.599474 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507313 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:26:31.599474 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507316 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:26:31.599474 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507319 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:26:31.599474 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507321 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:26:31.600302 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507324 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:26:31.600302 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507326 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:26:31.600302 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507329 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:26:31.600302 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507331 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:26:31.600302 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507333 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:26:31.600302 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507338 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:26:31.600302 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507341 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:26:31.600302 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507344 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:26:31.600302 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507346 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:26:31.600302 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507349 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:26:31.600302 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507353 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:26:31.600302 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507357 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:26:31.600302 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507360 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:26:31.600302 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507363 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:26:31.600302 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507366 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:26:31.600302 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507369 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:26:31.600302 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507371 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:26:31.600302 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507374 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:26:31.600302 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507377 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:26:31.601154 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507381 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:26:31.601154 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507384 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:26:31.601154 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507386 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:26:31.601154 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507389 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:26:31.601154 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507391 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:26:31.601154 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507395 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:26:31.601154 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507398 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:26:31.601154 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507400 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:26:31.601154 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507404 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:26:31.601154 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507408 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:26:31.601154 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507410 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:26:31.601154 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507413 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:26:31.601154 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507415 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:26:31.601154 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507418 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:26:31.601154 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507420 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:26:31.601154 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507423 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:26:31.601154 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507425 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:26:31.601154 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507428 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:26:31.601154 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507432 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:26:31.601154 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507434 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:26:31.601945 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507437 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:26:31.601945 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507440 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:26:31.601945 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507443 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:26:31.601945 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507445 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:26:31.601945 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507449 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:26:31.601945 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507452 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:26:31.601945 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507455 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:26:31.601945 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507458 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:26:31.601945 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507475 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:26:31.601945 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507478 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:26:31.601945 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507481 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:26:31.601945 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507484 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:26:31.601945 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507488 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:26:31.601945 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507491 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:26:31.601945 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507494 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:26:31.601945 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507496 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:26:31.601945 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507499 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:26:31.601945 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507501 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:26:31.601945 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507504 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:26:31.601945 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507507 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:26:31.602673 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507510 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:26:31.602673 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507513 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:26:31.602673 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507515 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:26:31.602673 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507518 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:26:31.602673 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507520 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:26:31.602673 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507523 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:26:31.602673 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507525 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:26:31.602673 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507528 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:26:31.602673 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507530 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:26:31.602673 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507532 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:26:31.602673 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507537 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:26:31.602673 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507540 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:26:31.602673 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507542 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:26:31.602673 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507545 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:26:31.602673 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.507547 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:26:31.602673 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507608 2575 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 19:26:31.602673 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507615 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 19:26:31.602673 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507622 2575 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 19:26:31.602673 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507637 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 19:26:31.602673 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507640 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 19:26:31.602673 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507643 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 19:26:31.603624 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507648 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 19:26:31.603624 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507651 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 19:26:31.603624 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507655 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 19:26:31.603624 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507658 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 19:26:31.603624 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507661 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 19:26:31.603624 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507664 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 19:26:31.603624 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507668 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 19:26:31.603624 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507671 2575 flags.go:64] FLAG: --cgroup-root="" Jul 06 19:26:31.603624 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507673 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 19:26:31.603624 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507676 2575 flags.go:64] FLAG: --client-ca-file="" Jul 06 19:26:31.603624 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507680 2575 flags.go:64] FLAG: --cloud-provider="external" Jul 06 19:26:31.603624 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507683 2575 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 19:26:31.603624 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507687 2575 flags.go:64] FLAG: --cluster-domain="" Jul 06 19:26:31.603624 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507690 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 19:26:31.603624 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507693 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 19:26:31.603624 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507696 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 19:26:31.603624 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507699 2575 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 19:26:31.603624 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507702 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 19:26:31.603624 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507705 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 19:26:31.603624 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507708 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 19:26:31.603624 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507712 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 19:26:31.603624 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507716 2575 flags.go:64] FLAG: --contention-profiling="false" Jul 06 19:26:31.603624 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507719 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 19:26:31.603624 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507722 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 19:26:31.605700 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507725 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 19:26:31.605700 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507728 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 19:26:31.605700 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507732 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 19:26:31.605700 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507735 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 19:26:31.605700 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507738 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 19:26:31.605700 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507740 2575 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 19:26:31.605700 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507743 2575 flags.go:64] FLAG: --enable-server="true" Jul 06 19:26:31.605700 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507746 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 19:26:31.605700 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507751 2575 flags.go:64] FLAG: --event-burst="100" Jul 06 19:26:31.605700 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507754 2575 flags.go:64] FLAG: --event-qps="50" Jul 06 19:26:31.605700 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507757 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 19:26:31.605700 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507760 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 19:26:31.605700 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507763 2575 flags.go:64] FLAG: --eviction-hard="" Jul 06 19:26:31.605700 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507768 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 19:26:31.605700 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507771 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 19:26:31.605700 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507773 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 19:26:31.605700 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507776 2575 flags.go:64] FLAG: --eviction-soft="" Jul 06 19:26:31.605700 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507779 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 19:26:31.605700 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507782 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 19:26:31.605700 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507785 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 19:26:31.605700 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507788 2575 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 19:26:31.605700 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507791 2575 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 19:26:31.605700 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507794 2575 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 19:26:31.605700 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507797 2575 flags.go:64] FLAG: --feature-gates="" Jul 06 19:26:31.605700 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507800 2575 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 19:26:31.606692 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507803 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 19:26:31.606692 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507806 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 19:26:31.606692 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507809 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 19:26:31.606692 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507812 2575 flags.go:64] FLAG: --healthz-port="10248" Jul 06 19:26:31.606692 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507816 2575 flags.go:64] FLAG: --help="false" Jul 06 19:26:31.606692 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507819 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-135-23.ec2.internal" Jul 06 19:26:31.606692 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507822 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 19:26:31.606692 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507825 2575 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 19:26:31.606692 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507828 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 19:26:31.606692 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507831 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 19:26:31.606692 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507834 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 19:26:31.606692 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507837 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 19:26:31.606692 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507839 2575 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 19:26:31.606692 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507842 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 19:26:31.606692 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507845 2575 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 19:26:31.606692 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507848 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 19:26:31.606692 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507851 2575 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 19:26:31.606692 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507854 2575 flags.go:64] FLAG: --kube-reserved="" Jul 06 19:26:31.606692 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507856 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 19:26:31.606692 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507859 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 19:26:31.606692 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507863 2575 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 19:26:31.606692 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507866 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 19:26:31.606692 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507869 2575 flags.go:64] FLAG: --lock-file="" Jul 06 19:26:31.606692 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507871 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 19:26:31.607719 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507874 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 19:26:31.607719 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507877 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 19:26:31.607719 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507882 2575 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 19:26:31.607719 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507886 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 19:26:31.607719 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507889 2575 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 19:26:31.607719 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507891 2575 flags.go:64] FLAG: --logging-format="text" Jul 06 19:26:31.607719 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507894 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 19:26:31.607719 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507897 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 19:26:31.607719 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507900 2575 flags.go:64] FLAG: --manifest-url="" Jul 06 19:26:31.607719 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507903 2575 flags.go:64] FLAG: --manifest-url-header="" Jul 06 19:26:31.607719 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507906 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 19:26:31.607719 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507909 2575 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 19:26:31.607719 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507914 2575 flags.go:64] FLAG: --max-pods="110" Jul 06 19:26:31.607719 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507917 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 19:26:31.607719 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507920 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 19:26:31.607719 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507923 2575 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 19:26:31.607719 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507926 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 19:26:31.607719 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507929 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 19:26:31.607719 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507931 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 19:26:31.607719 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507934 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 19:26:31.607719 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507941 2575 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 19:26:31.607719 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507944 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 19:26:31.607719 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507947 2575 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 19:26:31.607719 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507950 2575 flags.go:64] FLAG: --pod-cidr="" Jul 06 19:26:31.608509 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507953 2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 19:26:31.608509 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507958 2575 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 19:26:31.608509 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507961 2575 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 19:26:31.608509 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507964 2575 flags.go:64] FLAG: --pods-per-core="0" Jul 06 19:26:31.608509 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507968 2575 flags.go:64] FLAG: --port="10250" Jul 06 19:26:31.608509 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507971 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 19:26:31.608509 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507974 2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-029701e4fb8ee3da0" Jul 06 19:26:31.608509 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507977 2575 flags.go:64] FLAG: --qos-reserved="" Jul 06 19:26:31.608509 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507980 2575 flags.go:64] FLAG: --read-only-port="10255" Jul 06 19:26:31.608509 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507983 2575 flags.go:64] FLAG: --register-node="true" Jul 06 19:26:31.608509 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507985 2575 flags.go:64] FLAG: --register-with-taints="" Jul 06 19:26:31.608509 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507989 2575 flags.go:64] FLAG: --registry-burst="10" Jul 06 19:26:31.608509 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507992 2575 flags.go:64] FLAG: --registry-qps="5" Jul 06 19:26:31.608509 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507995 2575 flags.go:64] FLAG: --reserved-cpus="" Jul 06 19:26:31.608509 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.507998 2575 flags.go:64] FLAG: --reserved-memory="" Jul 06 19:26:31.608509 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.508001 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 19:26:31.608509 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.508004 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 19:26:31.608509 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.508007 2575 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 19:26:31.608509 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.508010 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 19:26:31.608509 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.508013 2575 flags.go:64] FLAG: --runonce="false" Jul 06 19:26:31.608509 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.508015 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 19:26:31.608509 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.508019 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 19:26:31.608509 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.508022 2575 flags.go:64] FLAG: --seccomp-default="false" Jul 06 19:26:31.608509 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.508025 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 19:26:31.608509 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.508028 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 19:26:31.609214 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.508031 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 19:26:31.609214 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.508033 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 19:26:31.609214 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.508036 2575 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 19:26:31.609214 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.508039 2575 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 19:26:31.609214 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.508042 2575 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 19:26:31.609214 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.508044 2575 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 19:26:31.609214 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.508047 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 19:26:31.609214 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.508050 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 19:26:31.609214 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.508053 2575 flags.go:64] FLAG: --system-cgroups="" Jul 06 19:26:31.609214 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.508056 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 19:26:31.609214 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.508061 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 19:26:31.609214 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.508064 2575 flags.go:64] FLAG: --tls-cert-file="" Jul 06 19:26:31.609214 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.508067 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 19:26:31.609214 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.508072 2575 flags.go:64] FLAG: --tls-min-version="" Jul 06 19:26:31.609214 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.508074 2575 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 19:26:31.609214 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.508077 2575 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 19:26:31.609214 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.508080 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 19:26:31.609214 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.508083 2575 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 19:26:31.609214 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.508086 2575 flags.go:64] FLAG: --v="2" Jul 06 19:26:31.609214 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.508090 2575 flags.go:64] FLAG: --version="false" Jul 06 19:26:31.609214 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.508094 2575 flags.go:64] FLAG: --vmodule="" Jul 06 19:26:31.609214 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.508098 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 19:26:31.609214 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.508101 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 19:26:31.609214 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508224 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:26:31.609214 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508228 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:26:31.609971 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508231 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:26:31.609971 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508234 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:26:31.609971 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508237 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:26:31.609971 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508243 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:26:31.609971 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508247 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:26:31.609971 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508250 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:26:31.609971 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508253 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:26:31.609971 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508256 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:26:31.609971 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508259 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:26:31.609971 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508262 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:26:31.609971 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508264 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:26:31.609971 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508267 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:26:31.609971 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508270 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:26:31.609971 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508272 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:26:31.609971 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508274 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:26:31.609971 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508278 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:26:31.609971 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508280 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:26:31.609971 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508283 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:26:31.609971 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508287 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:26:31.610609 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508289 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:26:31.610609 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508292 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:26:31.610609 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508294 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:26:31.610609 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508297 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:26:31.610609 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508299 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:26:31.610609 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508302 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:26:31.610609 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508304 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:26:31.610609 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508307 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:26:31.610609 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508310 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:26:31.610609 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508313 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:26:31.610609 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508316 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:26:31.610609 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508318 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:26:31.610609 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508321 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:26:31.610609 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508323 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:26:31.610609 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508326 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:26:31.610609 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508330 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:26:31.610609 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508335 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:26:31.610609 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508338 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:26:31.610609 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508341 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:26:31.802397 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508344 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:26:31.802397 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508347 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:26:31.802397 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508350 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:26:31.802397 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508354 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:26:31.802397 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508357 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:26:31.802397 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508360 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:26:31.802397 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508362 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:26:31.802397 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508365 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:26:31.802397 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508367 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:26:31.802397 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508370 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:26:31.802397 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508372 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:26:31.802397 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508375 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:26:31.802397 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508379 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:26:31.802397 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508381 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:26:31.802397 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508384 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:26:31.802397 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508386 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:26:31.802397 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508389 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:26:31.802397 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508391 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:26:31.802397 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508394 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:26:31.818330 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508396 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:26:31.818330 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508399 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:26:31.818330 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508401 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:26:31.818330 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508404 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:26:31.818330 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508407 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:26:31.818330 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508410 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:26:31.818330 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508412 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:26:31.818330 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508415 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:26:31.818330 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508417 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:26:31.818330 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508419 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:26:31.818330 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508423 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:26:31.818330 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508426 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:26:31.818330 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508429 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:26:31.818330 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508431 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:26:31.818330 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508433 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:26:31.818330 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508436 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:26:31.818330 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508438 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:26:31.818330 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508441 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:26:31.818330 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508443 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:26:31.818330 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508446 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:26:31.818952 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508448 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:26:31.818952 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508451 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:26:31.818952 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508453 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:26:31.818952 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508456 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:26:31.818952 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508458 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:26:31.818952 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508462 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:26:31.818952 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508464 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:26:31.818952 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508467 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:26:31.818952 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508469 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:26:31.818952 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508472 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:26:31.818952 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508474 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:26:31.818952 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508477 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:26:31.818952 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508479 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:26:31.818952 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508482 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:26:31.818952 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508484 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:26:31.818952 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508487 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:26:31.818952 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508489 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:26:31.818952 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508492 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:26:31.818952 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508495 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:26:31.818952 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.508497 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:26:31.819645 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.509050 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 19:26:31.819645 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.515636 2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 19:26:31.819645 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.515649 2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 19:26:31.819645 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.515668 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 19:26:31.819645 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.515673 2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 19:26:31.819645 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515724 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:26:31.819645 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515729 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:26:31.819645 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515733 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:26:31.819645 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515736 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:26:31.819645 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515739 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:26:31.819645 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515742 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:26:31.819645 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515745 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:26:31.819645 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515748 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:26:31.819645 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515751 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:26:31.819645 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515754 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:26:31.820155 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515756 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:26:31.820155 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515759 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:26:31.820155 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515762 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:26:31.820155 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515764 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:26:31.820155 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515767 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:26:31.820155 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515770 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:26:31.820155 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515773 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:26:31.820155 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515775 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:26:31.820155 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515778 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:26:31.820155 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515780 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:26:31.820155 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515783 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:26:31.820155 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515786 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:26:31.820155 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515789 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:26:31.820155 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515791 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:26:31.820155 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515795 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:26:31.820155 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515798 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:26:31.820155 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515801 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:26:31.820155 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515804 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:26:31.820155 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515806 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:26:31.820155 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515809 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:26:31.820748 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515811 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:26:31.820748 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515814 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:26:31.820748 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515817 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:26:31.820748 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515819 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:26:31.820748 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515824 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:26:31.820748 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515827 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:26:31.820748 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515830 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:26:31.820748 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515832 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:26:31.820748 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515835 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:26:31.820748 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515838 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:26:31.820748 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515841 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:26:31.820748 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515843 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:26:31.820748 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515846 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:26:31.820748 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515848 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:26:31.820748 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515851 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:26:31.820748 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515853 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:26:31.820748 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515856 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:26:31.820748 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515859 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:26:31.820748 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515862 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:26:31.821341 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515865 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:26:31.821341 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515869 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:26:31.821341 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515872 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:26:31.821341 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515875 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:26:31.821341 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515878 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:26:31.821341 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515881 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:26:31.821341 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515884 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:26:31.821341 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515887 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:26:31.821341 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515890 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:26:31.821341 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515894 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:26:31.821341 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515898 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:26:31.821341 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515901 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:26:31.821341 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515904 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:26:31.821341 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515906 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:26:31.821341 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515909 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:26:31.821341 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515912 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:26:31.821341 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515914 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:26:31.821341 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515918 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:26:31.821341 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515920 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:26:31.821889 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515925 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:26:31.821889 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515928 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:26:31.821889 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515931 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:26:31.821889 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515934 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:26:31.821889 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515936 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:26:31.821889 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515939 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:26:31.821889 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515941 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:26:31.821889 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515944 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:26:31.821889 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515947 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:26:31.821889 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515950 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:26:31.821889 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515953 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:26:31.821889 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515955 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:26:31.821889 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515970 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:26:31.821889 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515974 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:26:31.821889 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515977 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:26:31.821889 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515980 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:26:31.821889 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515983 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:26:31.821889 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515986 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:26:31.821889 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515989 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:26:31.821889 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515992 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:26:31.822513 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515995 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:26:31.822513 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.515998 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:26:31.822513 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516001 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:26:31.822513 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516004 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:26:31.822513 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516007 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:26:31.822513 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516009 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:26:31.822513 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516012 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:26:31.822513 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516014 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:26:31.822513 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516017 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:26:31.822513 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516020 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:26:31.822513 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516022 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:26:31.822513 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.516027 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 19:26:31.822513 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516134 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:26:31.822513 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516137 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:26:31.822513 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516140 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:26:31.822965 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516143 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:26:31.822965 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516146 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:26:31.822965 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516149 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:26:31.822965 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516152 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:26:31.822965 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516155 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:26:31.822965 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516157 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:26:31.822965 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516160 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:26:31.822965 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516163 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:26:31.822965 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516166 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:26:31.822965 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516169 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:26:31.822965 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516173 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:26:31.822965 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516176 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:26:31.822965 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516194 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:26:31.822965 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516198 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:26:31.822965 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516201 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:26:31.822965 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516203 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:26:31.822965 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516206 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:26:31.822965 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516209 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:26:31.822965 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516211 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:26:31.823978 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516214 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:26:31.823978 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516217 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:26:31.823978 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516220 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:26:31.823978 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516222 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:26:31.823978 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516225 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:26:31.823978 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516228 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:26:31.823978 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516230 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:26:31.823978 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516234 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:26:31.823978 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516236 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:26:31.823978 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516239 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:26:31.823978 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516243 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:26:31.823978 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516245 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:26:31.823978 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516250 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:26:31.823978 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516252 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:26:31.823978 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516255 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:26:31.823978 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516258 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:26:31.823978 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516260 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:26:31.823978 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516263 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:26:31.823978 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516265 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:26:31.823978 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516268 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:26:31.824805 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516270 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:26:31.824805 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516273 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:26:31.824805 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516276 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:26:31.824805 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516279 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:26:31.824805 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516281 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:26:31.824805 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516284 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:26:31.824805 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516286 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:26:31.824805 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516289 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:26:31.824805 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516291 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:26:31.824805 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516295 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:26:31.824805 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516298 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:26:31.824805 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516301 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:26:31.824805 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516304 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:26:31.824805 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516307 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:26:31.824805 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516313 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:26:31.824805 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516316 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:26:31.824805 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516318 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:26:31.824805 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516321 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:26:31.824805 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516324 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:26:31.825796 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516326 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:26:31.825796 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516329 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:26:31.825796 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516332 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:26:31.825796 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516336 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:26:31.825796 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516338 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:26:31.825796 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516342 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:26:31.825796 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516344 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:26:31.825796 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516347 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:26:31.825796 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516350 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:26:31.825796 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516352 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:26:31.825796 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516355 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:26:31.825796 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516358 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:26:31.825796 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516362 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:26:31.825796 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516365 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:26:31.825796 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516367 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:26:31.825796 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516370 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:26:31.825796 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516372 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:26:31.825796 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516375 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:26:31.825796 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516377 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:26:31.826397 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516380 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:26:31.826397 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516382 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:26:31.826397 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516385 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:26:31.826397 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516387 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:26:31.826397 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516390 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:26:31.826397 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516393 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:26:31.826397 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516395 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:26:31.826397 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516397 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:26:31.826397 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516400 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:26:31.826397 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516403 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:26:31.826397 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516406 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:26:31.826397 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516408 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:26:31.826397 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516410 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:26:31.826397 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516413 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:26:31.826397 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516416 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:26:31.826397 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516418 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:26:31.826397 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516423 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:26:31.826397 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516425 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:26:31.826397 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:31.516430 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:26:31.827019 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.516435 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 19:26:31.827019 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.516525 2575 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 19:26:31.827019 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.519520 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 19:26:31.827019 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.521158 2575 server.go:1019] "Starting client certificate rotation" Jul 06 19:26:31.827019 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.521268 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 19:26:31.827019 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.521312 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 19:26:31.827019 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.523811 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 19:26:31.827019 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.527654 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 19:26:31.827019 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.539256 2575 log.go:25] "Validated CRI v1 runtime API" Jul 06 19:26:31.827019 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.541364 2575 log.go:25] "Validated CRI v1 image API" Jul 06 19:26:31.827019 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.542126 2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 19:26:31.827019 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.545074 2575 fs.go:135] Filesystem UUIDs: map[0247bd54-e5c4-49c1-83f6-5f48bf06053f:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 cc2f09a6-38dd-4278-8ea0-391d6143234e:/dev/nvme0n1p4] Jul 06 19:26:31.827435 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.545087 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 19:26:31.827505 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.551484 2575 manager.go:216] Machine: {Timestamp:2026-07-06 19:26:31.549968441 +0000 UTC m=+0.370057144 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097516 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec225cb4774aacb8acbeaa221f17f29f SystemUUID:ec225cb4-774a-acb8-acbe-aa221f17f29f BootID:dbf29b5f-2266-4306-b4f9-0bc7e1711347 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:1e:7b:25:ef:0b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:1e:7b:25:ef:0b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:5b:ff:66:33:bf Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 19:26:31.827505 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.551583 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 19:26:31.827505 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.551683 2575 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 19:26:31.827505 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.552237 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 19:26:31.827505 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.554747 2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 19:26:31.827505 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.554771 2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-23.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 19:26:31.827505 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.555381 2575 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 19:26:31.827790 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.555391 2575 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 19:26:31.827790 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.555399 2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 19:26:31.827790 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.555412 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 19:26:31.827790 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.556623 2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 19:26:31.827790 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.559678 2575 state_mem.go:36] "Initialized new in-memory state store" Jul 06 19:26:31.827790 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.559795 2575 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 19:26:31.827790 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.559827 2575 kubelet.go:486] "Attempting to sync node with API server" Jul 06 19:26:31.827790 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.559848 2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 19:26:31.827790 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.559870 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 19:26:31.827790 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.559884 2575 kubelet.go:398] "Adding apiserver pod source" Jul 06 19:26:31.827790 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.559896 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 19:26:31.827790 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.560280 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 19:26:31.827790 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.560300 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 19:26:31.827790 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.562591 2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 19:26:31.827790 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.564222 2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 19:26:31.827790 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.564249 2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 19:26:31.827790 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.565446 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 19:26:31.827790 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.565459 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 19:26:31.827790 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.565466 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 19:26:31.827790 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.565472 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 19:26:31.827790 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.565481 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 19:26:31.827790 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.565488 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 19:26:31.827790 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.565500 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 19:26:31.827790 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.565507 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 19:26:31.827790 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.565513 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 19:26:31.827790 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.565518 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 19:26:31.827790 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.565532 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 19:26:31.827790 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.565538 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 19:26:31.827790 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.566156 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 19:26:31.827790 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.566162 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 19:26:31.827790 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:31.569538 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 19:26:31.827790 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:31.569538 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-23.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 19:26:31.827790 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.569642 2575 server.go:1268] "Started kubelet" Jul 06 19:26:31.827790 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.569648 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-23.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 06 19:26:31.827790 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.569704 2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 19:26:31.827790 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.569732 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 19:26:31.827790 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.569802 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 19:26:31.827790 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.571797 2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 19:26:31.827790 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.573001 2575 server.go:310] "Adding debug handlers to kubelet server" Jul 06 19:26:31.828877 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:31.575750 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-23.ec2.internal.18bfc9ebb004eb46 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-23.ec2.internal,UID:ip-10-0-135-23.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-23.ec2.internal,},FirstTimestamp:2026-07-06 19:26:31.569615686 +0000 UTC m=+0.389704390,LastTimestamp:2026-07-06 19:26:31.569615686 +0000 UTC m=+0.389704390,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-23.ec2.internal,}" Jul 06 19:26:31.828877 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:31.577775 2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 19:26:31.828877 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.579245 2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 19:26:31.828877 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.579245 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 19:26:31.828877 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:31.579325 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-23.ec2.internal\" not found" Jul 06 19:26:31.828877 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.580111 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 19:26:31.828877 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.580114 2575 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 19:26:31.828877 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.580134 2575 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 19:26:31.828877 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.580212 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 19:26:31.828877 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.580227 2575 factory.go:55] Registering systemd factory Jul 06 19:26:31.828877 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.580235 2575 factory.go:223] Registration of the systemd container factory successfully Jul 06 19:26:31.828877 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.580342 2575 reconstruct.go:97] "Volume reconstruction finished" Jul 06 19:26:31.828877 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.580353 2575 reconciler.go:29] "Reconciler: start to sync state" Jul 06 19:26:31.828877 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.580438 2575 factory.go:153] Registering CRI-O factory Jul 06 19:26:31.828877 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.580448 2575 factory.go:223] Registration of the crio container factory successfully Jul 06 19:26:31.828877 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.580469 2575 factory.go:103] Registering Raw factory Jul 06 19:26:31.828877 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.580478 2575 manager.go:1196] Started watching for new ooms in manager Jul 06 19:26:31.828877 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.580760 2575 manager.go:349] Starting recovery of all containers Jul 06 19:26:31.828877 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:31.582086 2575 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-23.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 06 19:26:31.828877 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:31.582246 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 06 19:26:31.828877 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.585893 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r2vbd" Jul 06 19:26:31.828877 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.591866 2575 manager.go:354] Recovery completed Jul 06 19:26:31.828877 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.597648 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:26:31.828877 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.599602 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-23.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:26:31.828877 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.599633 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-23.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:26:31.828877 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.599648 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-23.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:26:31.828877 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.600317 2575 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 19:26:31.828877 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.600330 2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 19:26:31.828877 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.600335 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:26:31.828877 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.600347 2575 state_mem.go:36] "Initialized new in-memory state store" Jul 06 19:26:31.828877 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.601000 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r2vbd" Jul 06 19:26:31.828877 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.601235 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-23.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:26:31.829763 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.601258 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-23.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:26:31.829763 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.601271 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-23.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:26:31.829763 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.602228 2575 policy_none.go:49] "None policy: Start" Jul 06 19:26:31.829763 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.602243 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 19:26:31.829763 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.602254 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 19:26:31.829763 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.603800 2575 policy_none.go:47] "Start" Jul 06 19:26:31.829763 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.652913 2575 manager.go:337] "Starting Device Plugin manager" Jul 06 19:26:31.829763 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:31.652947 2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 19:26:31.829763 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.652956 2575 server.go:87] "Starting device plugin registration server" Jul 06 19:26:31.829763 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.653204 2575 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 19:26:31.829763 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.653214 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 19:26:31.829763 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.653418 2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 19:26:31.829763 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.653493 2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 19:26:31.829763 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.653502 2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 19:26:31.829763 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:31.656029 2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 19:26:31.829763 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:31.656081 2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-23.ec2.internal\" not found" Jul 06 19:26:31.829763 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.753324 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:26:31.829763 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.754174 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-23.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:26:31.829763 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.754217 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-23.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:26:31.829763 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.754227 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-23.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:26:31.829763 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.754256 2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-23.ec2.internal" Jul 06 19:26:31.829763 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.766115 2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-23.ec2.internal" Jul 06 19:26:31.829763 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:31.766133 2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-23.ec2.internal\": node \"ip-10-0-135-23.ec2.internal\" not found" Jul 06 19:26:31.829763 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:31.785300 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-23.ec2.internal\" not found" Jul 06 19:26:31.829763 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.786522 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 19:26:31.829763 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.787745 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 19:26:31.829763 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.787790 2575 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 19:26:31.829763 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.787812 2575 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 19:26:31.829763 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:31.787849 2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 06 19:26:31.829763 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.789930 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:26:32.061249 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:31.885857 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-23.ec2.internal\" not found" Jul 06 19:26:32.061249 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.887929 2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-23.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-23.ec2.internal"] Jul 06 19:26:32.061249 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.888006 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:26:32.061249 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.889676 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-23.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:26:32.061249 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.889702 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-23.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:26:32.061249 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.889712 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-23.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:26:32.061249 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.890946 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:26:32.061249 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.891066 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-23.ec2.internal" Jul 06 19:26:32.061249 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.891106 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:26:32.061249 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.891717 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-23.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:26:32.061249 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.891748 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-23.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:26:32.061249 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.891760 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-23.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:26:32.061249 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.891720 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-23.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:26:32.061249 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.891821 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-23.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:26:32.061249 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.891833 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-23.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:26:32.061249 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.893086 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-23.ec2.internal" Jul 06 19:26:32.061249 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.893108 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:26:32.061249 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.893799 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-23.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:26:32.061249 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.893819 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-23.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:26:32.061249 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.893828 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-23.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:26:32.061249 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:31.925098 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-23.ec2.internal\" not found" node="ip-10-0-135-23.ec2.internal" Jul 06 19:26:32.061249 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:31.930078 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-23.ec2.internal\" not found" node="ip-10-0-135-23.ec2.internal" Jul 06 19:26:32.061249 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.982768 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b19b38d7a6af7f5567994a3fcadd5e98-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-23.ec2.internal\" (UID: \"b19b38d7a6af7f5567994a3fcadd5e98\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-23.ec2.internal" Jul 06 19:26:32.061249 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.982795 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b19b38d7a6af7f5567994a3fcadd5e98-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-23.ec2.internal\" (UID: \"b19b38d7a6af7f5567994a3fcadd5e98\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-23.ec2.internal" Jul 06 19:26:32.061249 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:31.982833 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/42d465f5997f682b6201edde58cb1d3a-config\") pod \"kube-apiserver-proxy-ip-10-0-135-23.ec2.internal\" (UID: \"42d465f5997f682b6201edde58cb1d3a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-23.ec2.internal" Jul 06 19:26:32.061249 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:31.986383 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-23.ec2.internal\" not found" Jul 06 19:26:32.270873 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:32.083222 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b19b38d7a6af7f5567994a3fcadd5e98-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-23.ec2.internal\" (UID: \"b19b38d7a6af7f5567994a3fcadd5e98\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-23.ec2.internal" Jul 06 19:26:32.270873 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:32.083259 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b19b38d7a6af7f5567994a3fcadd5e98-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-23.ec2.internal\" (UID: \"b19b38d7a6af7f5567994a3fcadd5e98\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-23.ec2.internal" Jul 06 19:26:32.270873 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:32.083282 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/42d465f5997f682b6201edde58cb1d3a-config\") pod \"kube-apiserver-proxy-ip-10-0-135-23.ec2.internal\" (UID: \"42d465f5997f682b6201edde58cb1d3a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-23.ec2.internal" Jul 06 19:26:32.270873 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:32.083326 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/42d465f5997f682b6201edde58cb1d3a-config\") pod \"kube-apiserver-proxy-ip-10-0-135-23.ec2.internal\" (UID: \"42d465f5997f682b6201edde58cb1d3a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-23.ec2.internal" Jul 06 19:26:32.270873 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:32.083336 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b19b38d7a6af7f5567994a3fcadd5e98-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-23.ec2.internal\" (UID: \"b19b38d7a6af7f5567994a3fcadd5e98\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-23.ec2.internal" Jul 06 19:26:32.270873 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:32.083345 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b19b38d7a6af7f5567994a3fcadd5e98-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-23.ec2.internal\" (UID: \"b19b38d7a6af7f5567994a3fcadd5e98\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-23.ec2.internal" Jul 06 19:26:32.270873 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:32.087292 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-23.ec2.internal\" not found" Jul 06 19:26:32.270873 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:32.187832 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-23.ec2.internal\" not found" Jul 06 19:26:32.270873 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:32.228987 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-23.ec2.internal" Jul 06 19:26:32.270873 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:32.234156 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-23.ec2.internal" Jul 06 19:26:32.498475 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:32.288501 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-23.ec2.internal\" not found" Jul 06 19:26:32.498475 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:32.389018 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-23.ec2.internal\" not found" Jul 06 19:26:32.498475 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:32.488729 2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:26:32.498475 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:32.489205 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-23.ec2.internal\" not found" Jul 06 19:26:32.675059 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:32.520541 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 19:26:32.675059 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:32.520679 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 19:26:32.675059 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:32.520710 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 19:26:32.675059 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:32.589343 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-23.ec2.internal\" not found" Jul 06 19:26:32.675059 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:32.603790 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 19:21:31 +0000 UTC" deadline="2028-04-06 14:40:46.544904628 +0000 UTC" Jul 06 19:26:32.675059 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:32.603812 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15355h14m13.941095065s" Jul 06 19:26:32.880632 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:32.689409 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-23.ec2.internal\" not found" Jul 06 19:26:32.880632 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:32.709274 2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:26:32.880632 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:32.789493 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-23.ec2.internal\" not found" Jul 06 19:26:32.880632 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:32.802814 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d465f5997f682b6201edde58cb1d3a.slice/crio-bfab5898f0c9d04da5a5d14de54a3557e95e23abcdc37891eb4cd5d48a97e258 WatchSource:0}: Error finding container bfab5898f0c9d04da5a5d14de54a3557e95e23abcdc37891eb4cd5d48a97e258: Status 404 returned error can't find the container with id bfab5898f0c9d04da5a5d14de54a3557e95e23abcdc37891eb4cd5d48a97e258 Jul 06 19:26:32.880632 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:32.806761 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:26:33.130127 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:32.890249 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-23.ec2.internal\" not found" Jul 06 19:26:33.130127 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:32.951443 2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:26:33.130127 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:32.980298 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-23.ec2.internal" Jul 06 19:26:33.130127 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:32.996634 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 19:26:33.130127 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:32.997888 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-23.ec2.internal" Jul 06 19:26:33.130127 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.010057 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 19:26:33.561325 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.561247 2575 apiserver.go:52] "Watching apiserver" Jul 06 19:26:33.572341 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.572297 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-h7z7k","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-23.ec2.internal","openshift-multus/network-metrics-daemon-ptml7","openshift-network-diagnostics/network-check-target-x5xtb","openshift-network-operator/iptables-alerter-6px9t","openshift-ovn-kubernetes/ovnkube-node-965cw","kube-system/konnectivity-agent-pgh76","kube-system/kube-apiserver-proxy-ip-10-0-135-23.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qndvc","openshift-cluster-node-tuning-operator/tuned-xlpxn","openshift-dns/node-resolver-2xwfj","openshift-image-registry/node-ca-q2tnp","openshift-multus/multus-additional-cni-plugins-ttjjl","openshift-multus/multus-dw4z9"] Jul 06 19:26:33.572341 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.572337 2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 19:26:33.574514 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.574490 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" Jul 06 19:26:33.575866 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.575845 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ttjjl" Jul 06 19:26:33.577373 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.577330 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bjtmx\"" Jul 06 19:26:33.577577 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.577563 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 19:26:33.577777 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.577762 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:26:33.577983 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.577964 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ptml7" Jul 06 19:26:33.578082 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:33.578034 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ptml7" podUID="c2f29af8-1539-48c2-a77e-3b7691f5cea8" Jul 06 19:26:33.578431 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.578397 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 19:26:33.578741 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.578718 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-j6q6c\"" Jul 06 19:26:33.579036 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.579019 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 19:26:33.579263 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.579246 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 19:26:33.579347 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.579330 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 19:26:33.579702 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.579687 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 19:26:33.579841 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.579826 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 19:26:33.579897 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.579868 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5xtb" Jul 06 19:26:33.579994 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:33.579970 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5xtb" podUID="867a102b-d74b-4541-8505-4f22e0871dd1" Jul 06 19:26:33.581045 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.581027 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6px9t" Jul 06 19:26:33.582930 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.582908 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.583672 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.583607 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:26:33.583795 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.583742 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 19:26:33.583795 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.583695 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-k5wcw\"" Jul 06 19:26:33.583930 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.583817 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 19:26:33.584691 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.584672 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.585603 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.585411 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 19:26:33.586029 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.585981 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 19:26:33.586109 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.586033 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 19:26:33.586927 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.586357 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qtdsm\"" Jul 06 19:26:33.586927 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.586492 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 19:26:33.586927 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.586678 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 19:26:33.586927 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.586684 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 19:26:33.586927 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.586729 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q2tnp" Jul 06 19:26:33.587346 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.587140 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 19:26:33.587611 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.587560 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jt8s4\"" Jul 06 19:26:33.587895 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.587879 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7z7k" Jul 06 19:26:33.587981 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:33.587954 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7z7k" podUID="696eee70-dc27-4b63-b656-ab71bd2ebec5" Jul 06 19:26:33.589214 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.589195 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2xwfj" Jul 06 19:26:33.589845 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.589667 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 19:26:33.589845 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.589706 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jwdjk\"" Jul 06 19:26:33.589995 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.589961 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 19:26:33.590049 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.590025 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 19:26:33.590560 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.590538 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc44b87e-83bc-4d47-b262-120b31e7e65c-etc-kubernetes\") pod \"tuned-xlpxn\" (UID: \"dc44b87e-83bc-4d47-b262-120b31e7e65c\") " pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" Jul 06 19:26:33.590641 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.590622 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/189cc7dd-7ba3-4cfc-b51c-4a8ab328f621-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ttjjl\" (UID: \"189cc7dd-7ba3-4cfc-b51c-4a8ab328f621\") " pod="openshift-multus/multus-additional-cni-plugins-ttjjl" Jul 06 19:26:33.590709 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.590651 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dc44b87e-83bc-4d47-b262-120b31e7e65c-etc-systemd\") pod \"tuned-xlpxn\" (UID: \"dc44b87e-83bc-4d47-b262-120b31e7e65c\") " pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" Jul 06 19:26:33.590709 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.590673 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dc44b87e-83bc-4d47-b262-120b31e7e65c-run\") pod \"tuned-xlpxn\" (UID: \"dc44b87e-83bc-4d47-b262-120b31e7e65c\") " pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" Jul 06 19:26:33.590709 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.590697 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc44b87e-83bc-4d47-b262-120b31e7e65c-var-lib-kubelet\") pod \"tuned-xlpxn\" (UID: \"dc44b87e-83bc-4d47-b262-120b31e7e65c\") " pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" Jul 06 19:26:33.590853 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.590725 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/189cc7dd-7ba3-4cfc-b51c-4a8ab328f621-os-release\") pod \"multus-additional-cni-plugins-ttjjl\" (UID: \"189cc7dd-7ba3-4cfc-b51c-4a8ab328f621\") " pod="openshift-multus/multus-additional-cni-plugins-ttjjl" Jul 06 19:26:33.590853 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.590751 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxlbh\" (UniqueName: \"kubernetes.io/projected/189cc7dd-7ba3-4cfc-b51c-4a8ab328f621-kube-api-access-zxlbh\") pod \"multus-additional-cni-plugins-ttjjl\" (UID: \"189cc7dd-7ba3-4cfc-b51c-4a8ab328f621\") " pod="openshift-multus/multus-additional-cni-plugins-ttjjl" Jul 06 19:26:33.590853 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.590773 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8561ed5c-290c-44b6-8490-8256078f657b-host-slash\") pod \"iptables-alerter-6px9t\" (UID: \"8561ed5c-290c-44b6-8490-8256078f657b\") " pod="openshift-network-operator/iptables-alerter-6px9t" Jul 06 19:26:33.590853 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.590824 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dc44b87e-83bc-4d47-b262-120b31e7e65c-etc-sysctl-conf\") pod \"tuned-xlpxn\" (UID: \"dc44b87e-83bc-4d47-b262-120b31e7e65c\") " pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" Jul 06 19:26:33.590853 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.590851 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dc44b87e-83bc-4d47-b262-120b31e7e65c-etc-tuned\") pod \"tuned-xlpxn\" (UID: \"dc44b87e-83bc-4d47-b262-120b31e7e65c\") " pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" Jul 06 19:26:33.591095 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.590873 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dc44b87e-83bc-4d47-b262-120b31e7e65c-etc-sysctl-d\") pod \"tuned-xlpxn\" (UID: \"dc44b87e-83bc-4d47-b262-120b31e7e65c\") " pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" Jul 06 19:26:33.591095 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.590896 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc44b87e-83bc-4d47-b262-120b31e7e65c-sys\") pod \"tuned-xlpxn\" (UID: \"dc44b87e-83bc-4d47-b262-120b31e7e65c\") " pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" Jul 06 19:26:33.591095 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.590916 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dc44b87e-83bc-4d47-b262-120b31e7e65c-lib-modules\") pod \"tuned-xlpxn\" (UID: \"dc44b87e-83bc-4d47-b262-120b31e7e65c\") " pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" Jul 06 19:26:33.591095 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.590941 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hmnl\" (UniqueName: \"kubernetes.io/projected/c2f29af8-1539-48c2-a77e-3b7691f5cea8-kube-api-access-5hmnl\") pod \"network-metrics-daemon-ptml7\" (UID: \"c2f29af8-1539-48c2-a77e-3b7691f5cea8\") " pod="openshift-multus/network-metrics-daemon-ptml7" Jul 06 19:26:33.591095 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.590964 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/189cc7dd-7ba3-4cfc-b51c-4a8ab328f621-system-cni-dir\") pod \"multus-additional-cni-plugins-ttjjl\" (UID: \"189cc7dd-7ba3-4cfc-b51c-4a8ab328f621\") " pod="openshift-multus/multus-additional-cni-plugins-ttjjl" Jul 06 19:26:33.591095 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.590986 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/189cc7dd-7ba3-4cfc-b51c-4a8ab328f621-cnibin\") pod \"multus-additional-cni-plugins-ttjjl\" (UID: \"189cc7dd-7ba3-4cfc-b51c-4a8ab328f621\") " pod="openshift-multus/multus-additional-cni-plugins-ttjjl" Jul 06 19:26:33.591095 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.591010 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2f29af8-1539-48c2-a77e-3b7691f5cea8-metrics-certs\") pod \"network-metrics-daemon-ptml7\" (UID: \"c2f29af8-1539-48c2-a77e-3b7691f5cea8\") " pod="openshift-multus/network-metrics-daemon-ptml7" Jul 06 19:26:33.591095 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.591033 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dc44b87e-83bc-4d47-b262-120b31e7e65c-etc-modprobe-d\") pod \"tuned-xlpxn\" (UID: \"dc44b87e-83bc-4d47-b262-120b31e7e65c\") " pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" Jul 06 19:26:33.591485 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.591078 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc44b87e-83bc-4d47-b262-120b31e7e65c-tmp\") pod \"tuned-xlpxn\" (UID: \"dc44b87e-83bc-4d47-b262-120b31e7e65c\") " pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" Jul 06 19:26:33.591485 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.591121 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/189cc7dd-7ba3-4cfc-b51c-4a8ab328f621-cni-binary-copy\") pod \"multus-additional-cni-plugins-ttjjl\" (UID: \"189cc7dd-7ba3-4cfc-b51c-4a8ab328f621\") " pod="openshift-multus/multus-additional-cni-plugins-ttjjl" Jul 06 19:26:33.591485 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.591146 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/189cc7dd-7ba3-4cfc-b51c-4a8ab328f621-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ttjjl\" (UID: \"189cc7dd-7ba3-4cfc-b51c-4a8ab328f621\") " pod="openshift-multus/multus-additional-cni-plugins-ttjjl" Jul 06 19:26:33.591485 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.591169 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85pqf\" (UniqueName: \"kubernetes.io/projected/dc44b87e-83bc-4d47-b262-120b31e7e65c-kube-api-access-85pqf\") pod \"tuned-xlpxn\" (UID: \"dc44b87e-83bc-4d47-b262-120b31e7e65c\") " pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" Jul 06 19:26:33.591485 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.591212 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/189cc7dd-7ba3-4cfc-b51c-4a8ab328f621-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ttjjl\" (UID: \"189cc7dd-7ba3-4cfc-b51c-4a8ab328f621\") " pod="openshift-multus/multus-additional-cni-plugins-ttjjl" Jul 06 19:26:33.591485 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.591234 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dc44b87e-83bc-4d47-b262-120b31e7e65c-host\") pod \"tuned-xlpxn\" (UID: \"dc44b87e-83bc-4d47-b262-120b31e7e65c\") " pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" Jul 06 19:26:33.591485 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.591320 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk9j2\" (UniqueName: \"kubernetes.io/projected/867a102b-d74b-4541-8505-4f22e0871dd1-kube-api-access-hk9j2\") pod \"network-check-target-x5xtb\" (UID: \"867a102b-d74b-4541-8505-4f22e0871dd1\") " pod="openshift-network-diagnostics/network-check-target-x5xtb" Jul 06 19:26:33.591485 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.591348 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8561ed5c-290c-44b6-8490-8256078f657b-iptables-alerter-script\") pod \"iptables-alerter-6px9t\" (UID: \"8561ed5c-290c-44b6-8490-8256078f657b\") " pod="openshift-network-operator/iptables-alerter-6px9t" Jul 06 19:26:33.591485 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.591372 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnwt8\" (UniqueName: \"kubernetes.io/projected/8561ed5c-290c-44b6-8490-8256078f657b-kube-api-access-rnwt8\") pod \"iptables-alerter-6px9t\" (UID: \"8561ed5c-290c-44b6-8490-8256078f657b\") " pod="openshift-network-operator/iptables-alerter-6px9t" Jul 06 19:26:33.591485 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.591402 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dc44b87e-83bc-4d47-b262-120b31e7e65c-etc-sysconfig\") pod \"tuned-xlpxn\" (UID: \"dc44b87e-83bc-4d47-b262-120b31e7e65c\") " pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" Jul 06 19:26:33.591485 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.590583 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pgh76" Jul 06 19:26:33.592735 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.592514 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qndvc" Jul 06 19:26:33.592735 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.592617 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-h2sqw\"" Jul 06 19:26:33.592876 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.592838 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 19:26:33.592876 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.592857 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 19:26:33.595344 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.595324 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 19:26:33.596012 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.595612 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 19:26:33.596012 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.595713 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fshrh\"" Jul 06 19:26:33.596012 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.595904 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 19:26:33.596214 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.596039 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 19:26:33.596214 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.596107 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-gb97f\"" Jul 06 19:26:33.597223 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.597209 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 19:26:33.599588 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.599571 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 19:26:33.630159 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.630132 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zcbbx" Jul 06 19:26:33.638610 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.638417 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zcbbx" Jul 06 19:26:33.646888 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.646696 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:26:33.681647 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.681626 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 19:26:33.691660 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.691635 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/189cc7dd-7ba3-4cfc-b51c-4a8ab328f621-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ttjjl\" (UID: \"189cc7dd-7ba3-4cfc-b51c-4a8ab328f621\") " pod="openshift-multus/multus-additional-cni-plugins-ttjjl" Jul 06 19:26:33.691781 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.691663 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dc44b87e-83bc-4d47-b262-120b31e7e65c-host\") pod \"tuned-xlpxn\" (UID: \"dc44b87e-83bc-4d47-b262-120b31e7e65c\") " pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" Jul 06 19:26:33.691781 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.691683 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/facd7b54-280f-4ee9-91df-3a0773f5dc0a-log-socket\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.691781 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.691699 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e6171d1-c8d8-4800-8998-44acce38b440-host-var-lib-kubelet\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.691781 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.691714 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht2bf\" (UniqueName: \"kubernetes.io/projected/9bd815c1-1206-431e-9d76-ba36f79833ac-kube-api-access-ht2bf\") pod \"node-resolver-2xwfj\" (UID: \"9bd815c1-1206-431e-9d76-ba36f79833ac\") " pod="openshift-dns/node-resolver-2xwfj" Jul 06 19:26:33.691781 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.691732 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8561ed5c-290c-44b6-8490-8256078f657b-iptables-alerter-script\") pod \"iptables-alerter-6px9t\" (UID: \"8561ed5c-290c-44b6-8490-8256078f657b\") " pod="openshift-network-operator/iptables-alerter-6px9t" Jul 06 19:26:33.691781 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.691756 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnwt8\" (UniqueName: \"kubernetes.io/projected/8561ed5c-290c-44b6-8490-8256078f657b-kube-api-access-rnwt8\") pod \"iptables-alerter-6px9t\" (UID: \"8561ed5c-290c-44b6-8490-8256078f657b\") " pod="openshift-network-operator/iptables-alerter-6px9t" Jul 06 19:26:33.691781 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.691775 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1e221582-b895-4724-8053-b72741d40132-socket-dir\") pod \"aws-ebs-csi-driver-node-qndvc\" (UID: \"1e221582-b895-4724-8053-b72741d40132\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qndvc" Jul 06 19:26:33.692118 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.691790 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/facd7b54-280f-4ee9-91df-3a0773f5dc0a-run-openvswitch\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.692118 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.691804 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/facd7b54-280f-4ee9-91df-3a0773f5dc0a-host-cni-netd\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.692118 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.691822 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4e6171d1-c8d8-4800-8998-44acce38b440-host-run-multus-certs\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.692118 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.691843 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/189cc7dd-7ba3-4cfc-b51c-4a8ab328f621-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ttjjl\" (UID: \"189cc7dd-7ba3-4cfc-b51c-4a8ab328f621\") " pod="openshift-multus/multus-additional-cni-plugins-ttjjl" Jul 06 19:26:33.692118 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.691870 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dc44b87e-83bc-4d47-b262-120b31e7e65c-etc-systemd\") pod \"tuned-xlpxn\" (UID: \"dc44b87e-83bc-4d47-b262-120b31e7e65c\") " pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" Jul 06 19:26:33.692118 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.691891 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e6171d1-c8d8-4800-8998-44acce38b440-multus-cni-dir\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.692118 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.691938 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e6171d1-c8d8-4800-8998-44acce38b440-cni-binary-copy\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.692118 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.691984 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e6171d1-c8d8-4800-8998-44acce38b440-host-run-netns\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.692118 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.691990 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dc44b87e-83bc-4d47-b262-120b31e7e65c-etc-systemd\") pod \"tuned-xlpxn\" (UID: \"dc44b87e-83bc-4d47-b262-120b31e7e65c\") " pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" Jul 06 19:26:33.692118 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.692007 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/556124af-4be3-4157-8b4b-70c1c1dfb6fb-konnectivity-ca\") pod \"konnectivity-agent-pgh76\" (UID: \"556124af-4be3-4157-8b4b-70c1c1dfb6fb\") " pod="kube-system/konnectivity-agent-pgh76" Jul 06 19:26:33.692118 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.692058 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1e221582-b895-4724-8053-b72741d40132-registration-dir\") pod \"aws-ebs-csi-driver-node-qndvc\" (UID: \"1e221582-b895-4724-8053-b72741d40132\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qndvc" Jul 06 19:26:33.692118 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.692071 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dc44b87e-83bc-4d47-b262-120b31e7e65c-host\") pod \"tuned-xlpxn\" (UID: \"dc44b87e-83bc-4d47-b262-120b31e7e65c\") " pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" Jul 06 19:26:33.692118 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.692093 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/facd7b54-280f-4ee9-91df-3a0773f5dc0a-host-slash\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.692118 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.692121 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/facd7b54-280f-4ee9-91df-3a0773f5dc0a-run-systemd\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.692763 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.692140 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4e6171d1-c8d8-4800-8998-44acce38b440-os-release\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.692763 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.692161 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4e6171d1-c8d8-4800-8998-44acce38b440-host-run-k8s-cni-cncf-io\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.692763 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.692203 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4e6171d1-c8d8-4800-8998-44acce38b440-host-var-lib-cni-multus\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.692763 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.692241 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hmnl\" (UniqueName: \"kubernetes.io/projected/c2f29af8-1539-48c2-a77e-3b7691f5cea8-kube-api-access-5hmnl\") pod \"network-metrics-daemon-ptml7\" (UID: \"c2f29af8-1539-48c2-a77e-3b7691f5cea8\") " pod="openshift-multus/network-metrics-daemon-ptml7" Jul 06 19:26:33.692763 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.692264 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1e221582-b895-4724-8053-b72741d40132-sys-fs\") pod \"aws-ebs-csi-driver-node-qndvc\" (UID: \"1e221582-b895-4724-8053-b72741d40132\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qndvc" Jul 06 19:26:33.692763 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.692365 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/189cc7dd-7ba3-4cfc-b51c-4a8ab328f621-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ttjjl\" (UID: \"189cc7dd-7ba3-4cfc-b51c-4a8ab328f621\") " pod="openshift-multus/multus-additional-cni-plugins-ttjjl" Jul 06 19:26:33.692763 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.692365 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8561ed5c-290c-44b6-8490-8256078f657b-iptables-alerter-script\") pod \"iptables-alerter-6px9t\" (UID: \"8561ed5c-290c-44b6-8490-8256078f657b\") " pod="openshift-network-operator/iptables-alerter-6px9t" Jul 06 19:26:33.692763 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.692402 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/facd7b54-280f-4ee9-91df-3a0773f5dc0a-run-ovn\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.692763 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.692418 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/189cc7dd-7ba3-4cfc-b51c-4a8ab328f621-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ttjjl\" (UID: \"189cc7dd-7ba3-4cfc-b51c-4a8ab328f621\") " pod="openshift-multus/multus-additional-cni-plugins-ttjjl" Jul 06 19:26:33.692763 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.692461 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/facd7b54-280f-4ee9-91df-3a0773f5dc0a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.692763 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.692488 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e6171d1-c8d8-4800-8998-44acce38b440-multus-conf-dir\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.692763 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.692515 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/863c8526-b822-48d4-8881-819331146578-host\") pod \"node-ca-q2tnp\" (UID: \"863c8526-b822-48d4-8881-819331146578\") " pod="openshift-image-registry/node-ca-q2tnp" Jul 06 19:26:33.692763 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.692538 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9bd815c1-1206-431e-9d76-ba36f79833ac-hosts-file\") pod \"node-resolver-2xwfj\" (UID: \"9bd815c1-1206-431e-9d76-ba36f79833ac\") " pod="openshift-dns/node-resolver-2xwfj" Jul 06 19:26:33.692763 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.692565 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/189cc7dd-7ba3-4cfc-b51c-4a8ab328f621-cnibin\") pod \"multus-additional-cni-plugins-ttjjl\" (UID: \"189cc7dd-7ba3-4cfc-b51c-4a8ab328f621\") " pod="openshift-multus/multus-additional-cni-plugins-ttjjl" Jul 06 19:26:33.693453 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.692589 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dc44b87e-83bc-4d47-b262-120b31e7e65c-etc-modprobe-d\") pod \"tuned-xlpxn\" (UID: \"dc44b87e-83bc-4d47-b262-120b31e7e65c\") " pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" Jul 06 19:26:33.693453 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.692622 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/facd7b54-280f-4ee9-91df-3a0773f5dc0a-host-kubelet\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.693453 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.692633 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/189cc7dd-7ba3-4cfc-b51c-4a8ab328f621-cnibin\") pod \"multus-additional-cni-plugins-ttjjl\" (UID: \"189cc7dd-7ba3-4cfc-b51c-4a8ab328f621\") " pod="openshift-multus/multus-additional-cni-plugins-ttjjl" Jul 06 19:26:33.693453 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.692668 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/facd7b54-280f-4ee9-91df-3a0773f5dc0a-var-lib-openvswitch\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.693453 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.692692 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e6171d1-c8d8-4800-8998-44acce38b440-cnibin\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.693453 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.692715 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n4cf\" (UniqueName: \"kubernetes.io/projected/4e6171d1-c8d8-4800-8998-44acce38b440-kube-api-access-4n4cf\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.693453 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.692738 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/189cc7dd-7ba3-4cfc-b51c-4a8ab328f621-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ttjjl\" (UID: \"189cc7dd-7ba3-4cfc-b51c-4a8ab328f621\") " pod="openshift-multus/multus-additional-cni-plugins-ttjjl" Jul 06 19:26:33.693453 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.692774 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85pqf\" (UniqueName: \"kubernetes.io/projected/dc44b87e-83bc-4d47-b262-120b31e7e65c-kube-api-access-85pqf\") pod \"tuned-xlpxn\" (UID: \"dc44b87e-83bc-4d47-b262-120b31e7e65c\") " pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" Jul 06 19:26:33.693453 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.692796 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dc44b87e-83bc-4d47-b262-120b31e7e65c-etc-modprobe-d\") pod \"tuned-xlpxn\" (UID: \"dc44b87e-83bc-4d47-b262-120b31e7e65c\") " pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" Jul 06 19:26:33.693453 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.692809 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1e221582-b895-4724-8053-b72741d40132-device-dir\") pod \"aws-ebs-csi-driver-node-qndvc\" (UID: \"1e221582-b895-4724-8053-b72741d40132\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qndvc" Jul 06 19:26:33.693453 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.692832 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/facd7b54-280f-4ee9-91df-3a0773f5dc0a-host-cni-bin\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.693453 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.692865 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/facd7b54-280f-4ee9-91df-3a0773f5dc0a-ovnkube-config\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.693453 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.692892 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4e6171d1-c8d8-4800-8998-44acce38b440-multus-socket-dir-parent\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.693453 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.692918 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1e221582-b895-4724-8053-b72741d40132-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qndvc\" (UID: \"1e221582-b895-4724-8053-b72741d40132\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qndvc" Jul 06 19:26:33.694046 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.692940 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4s5q\" (UniqueName: \"kubernetes.io/projected/facd7b54-280f-4ee9-91df-3a0773f5dc0a-kube-api-access-l4s5q\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.694046 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.692975 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqmpf\" (UniqueName: \"kubernetes.io/projected/863c8526-b822-48d4-8881-819331146578-kube-api-access-sqmpf\") pod \"node-ca-q2tnp\" (UID: \"863c8526-b822-48d4-8881-819331146578\") " pod="openshift-image-registry/node-ca-q2tnp" Jul 06 19:26:33.694046 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.692994 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/189cc7dd-7ba3-4cfc-b51c-4a8ab328f621-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ttjjl\" (UID: \"189cc7dd-7ba3-4cfc-b51c-4a8ab328f621\") " pod="openshift-multus/multus-additional-cni-plugins-ttjjl" Jul 06 19:26:33.694046 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.693040 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk9j2\" (UniqueName: \"kubernetes.io/projected/867a102b-d74b-4541-8505-4f22e0871dd1-kube-api-access-hk9j2\") pod \"network-check-target-x5xtb\" (UID: \"867a102b-d74b-4541-8505-4f22e0871dd1\") " pod="openshift-network-diagnostics/network-check-target-x5xtb" Jul 06 19:26:33.694046 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.693070 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dc44b87e-83bc-4d47-b262-120b31e7e65c-etc-sysconfig\") pod \"tuned-xlpxn\" (UID: \"dc44b87e-83bc-4d47-b262-120b31e7e65c\") " pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" Jul 06 19:26:33.694046 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.693096 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc44b87e-83bc-4d47-b262-120b31e7e65c-etc-kubernetes\") pod \"tuned-xlpxn\" (UID: \"dc44b87e-83bc-4d47-b262-120b31e7e65c\") " pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" Jul 06 19:26:33.694046 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.693138 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9txm\" (UniqueName: \"kubernetes.io/projected/1e221582-b895-4724-8053-b72741d40132-kube-api-access-f9txm\") pod \"aws-ebs-csi-driver-node-qndvc\" (UID: \"1e221582-b895-4724-8053-b72741d40132\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qndvc" Jul 06 19:26:33.694046 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.693150 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc44b87e-83bc-4d47-b262-120b31e7e65c-etc-kubernetes\") pod \"tuned-xlpxn\" (UID: \"dc44b87e-83bc-4d47-b262-120b31e7e65c\") " pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" Jul 06 19:26:33.694046 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.693168 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/facd7b54-280f-4ee9-91df-3a0773f5dc0a-ovn-node-metrics-cert\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.694046 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.693205 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dc44b87e-83bc-4d47-b262-120b31e7e65c-etc-sysconfig\") pod \"tuned-xlpxn\" (UID: \"dc44b87e-83bc-4d47-b262-120b31e7e65c\") " pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" Jul 06 19:26:33.694046 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.693207 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e6171d1-c8d8-4800-8998-44acce38b440-host-var-lib-cni-bin\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.694046 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.693254 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4e6171d1-c8d8-4800-8998-44acce38b440-hostroot\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.694046 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.693283 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dc44b87e-83bc-4d47-b262-120b31e7e65c-run\") pod \"tuned-xlpxn\" (UID: \"dc44b87e-83bc-4d47-b262-120b31e7e65c\") " pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" Jul 06 19:26:33.694046 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.693322 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/facd7b54-280f-4ee9-91df-3a0773f5dc0a-ovnkube-script-lib\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.694638 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.693337 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dc44b87e-83bc-4d47-b262-120b31e7e65c-run\") pod \"tuned-xlpxn\" (UID: \"dc44b87e-83bc-4d47-b262-120b31e7e65c\") " pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" Jul 06 19:26:33.694638 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.693351 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc44b87e-83bc-4d47-b262-120b31e7e65c-var-lib-kubelet\") pod \"tuned-xlpxn\" (UID: \"dc44b87e-83bc-4d47-b262-120b31e7e65c\") " pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" Jul 06 19:26:33.694638 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.693406 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/189cc7dd-7ba3-4cfc-b51c-4a8ab328f621-os-release\") pod \"multus-additional-cni-plugins-ttjjl\" (UID: \"189cc7dd-7ba3-4cfc-b51c-4a8ab328f621\") " pod="openshift-multus/multus-additional-cni-plugins-ttjjl" Jul 06 19:26:33.694638 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.693430 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxlbh\" (UniqueName: \"kubernetes.io/projected/189cc7dd-7ba3-4cfc-b51c-4a8ab328f621-kube-api-access-zxlbh\") pod \"multus-additional-cni-plugins-ttjjl\" (UID: \"189cc7dd-7ba3-4cfc-b51c-4a8ab328f621\") " pod="openshift-multus/multus-additional-cni-plugins-ttjjl" Jul 06 19:26:33.694638 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.693456 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8561ed5c-290c-44b6-8490-8256078f657b-host-slash\") pod \"iptables-alerter-6px9t\" (UID: \"8561ed5c-290c-44b6-8490-8256078f657b\") " pod="openshift-network-operator/iptables-alerter-6px9t" Jul 06 19:26:33.694638 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.693461 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc44b87e-83bc-4d47-b262-120b31e7e65c-var-lib-kubelet\") pod \"tuned-xlpxn\" (UID: \"dc44b87e-83bc-4d47-b262-120b31e7e65c\") " pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" Jul 06 19:26:33.694638 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.693479 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dc44b87e-83bc-4d47-b262-120b31e7e65c-etc-sysctl-conf\") pod \"tuned-xlpxn\" (UID: \"dc44b87e-83bc-4d47-b262-120b31e7e65c\") " pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" Jul 06 19:26:33.694638 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.693501 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dc44b87e-83bc-4d47-b262-120b31e7e65c-etc-tuned\") pod \"tuned-xlpxn\" (UID: \"dc44b87e-83bc-4d47-b262-120b31e7e65c\") " pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" Jul 06 19:26:33.694638 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.693507 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/189cc7dd-7ba3-4cfc-b51c-4a8ab328f621-os-release\") pod \"multus-additional-cni-plugins-ttjjl\" (UID: \"189cc7dd-7ba3-4cfc-b51c-4a8ab328f621\") " pod="openshift-multus/multus-additional-cni-plugins-ttjjl" Jul 06 19:26:33.694638 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.693510 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8561ed5c-290c-44b6-8490-8256078f657b-host-slash\") pod \"iptables-alerter-6px9t\" (UID: \"8561ed5c-290c-44b6-8490-8256078f657b\") " pod="openshift-network-operator/iptables-alerter-6px9t" Jul 06 19:26:33.694638 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.693523 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dc44b87e-83bc-4d47-b262-120b31e7e65c-etc-sysctl-d\") pod \"tuned-xlpxn\" (UID: \"dc44b87e-83bc-4d47-b262-120b31e7e65c\") " pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" Jul 06 19:26:33.694638 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.693629 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc44b87e-83bc-4d47-b262-120b31e7e65c-sys\") pod \"tuned-xlpxn\" (UID: \"dc44b87e-83bc-4d47-b262-120b31e7e65c\") " pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" Jul 06 19:26:33.694638 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.693634 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dc44b87e-83bc-4d47-b262-120b31e7e65c-etc-sysctl-conf\") pod \"tuned-xlpxn\" (UID: \"dc44b87e-83bc-4d47-b262-120b31e7e65c\") " pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" Jul 06 19:26:33.694638 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.693627 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dc44b87e-83bc-4d47-b262-120b31e7e65c-etc-sysctl-d\") pod \"tuned-xlpxn\" (UID: \"dc44b87e-83bc-4d47-b262-120b31e7e65c\") " pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" Jul 06 19:26:33.694638 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.693688 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dc44b87e-83bc-4d47-b262-120b31e7e65c-lib-modules\") pod \"tuned-xlpxn\" (UID: \"dc44b87e-83bc-4d47-b262-120b31e7e65c\") " pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" Jul 06 19:26:33.695229 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.693717 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/facd7b54-280f-4ee9-91df-3a0773f5dc0a-host-run-netns\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.695229 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.693743 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/facd7b54-280f-4ee9-91df-3a0773f5dc0a-etc-openvswitch\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.695229 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.693764 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc44b87e-83bc-4d47-b262-120b31e7e65c-sys\") pod \"tuned-xlpxn\" (UID: \"dc44b87e-83bc-4d47-b262-120b31e7e65c\") " pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" Jul 06 19:26:33.695229 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.693765 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/facd7b54-280f-4ee9-91df-3a0773f5dc0a-node-log\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.695229 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.693808 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/facd7b54-280f-4ee9-91df-3a0773f5dc0a-host-run-ovn-kubernetes\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.695229 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.693815 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dc44b87e-83bc-4d47-b262-120b31e7e65c-lib-modules\") pod \"tuned-xlpxn\" (UID: \"dc44b87e-83bc-4d47-b262-120b31e7e65c\") " pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" Jul 06 19:26:33.695229 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.693836 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e6171d1-c8d8-4800-8998-44acce38b440-system-cni-dir\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.695229 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.693838 2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 19:26:33.695229 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.693862 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e6171d1-c8d8-4800-8998-44acce38b440-etc-kubernetes\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.695229 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.693886 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/863c8526-b822-48d4-8881-819331146578-serviceca\") pod \"node-ca-q2tnp\" (UID: \"863c8526-b822-48d4-8881-819331146578\") " pod="openshift-image-registry/node-ca-q2tnp" Jul 06 19:26:33.695229 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.693914 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/189cc7dd-7ba3-4cfc-b51c-4a8ab328f621-system-cni-dir\") pod \"multus-additional-cni-plugins-ttjjl\" (UID: \"189cc7dd-7ba3-4cfc-b51c-4a8ab328f621\") " pod="openshift-multus/multus-additional-cni-plugins-ttjjl" Jul 06 19:26:33.695229 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.693944 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2f29af8-1539-48c2-a77e-3b7691f5cea8-metrics-certs\") pod \"network-metrics-daemon-ptml7\" (UID: \"c2f29af8-1539-48c2-a77e-3b7691f5cea8\") " pod="openshift-multus/network-metrics-daemon-ptml7" Jul 06 19:26:33.695229 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.693962 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/189cc7dd-7ba3-4cfc-b51c-4a8ab328f621-system-cni-dir\") pod \"multus-additional-cni-plugins-ttjjl\" (UID: \"189cc7dd-7ba3-4cfc-b51c-4a8ab328f621\") " pod="openshift-multus/multus-additional-cni-plugins-ttjjl" Jul 06 19:26:33.695229 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.693970 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc44b87e-83bc-4d47-b262-120b31e7e65c-tmp\") pod \"tuned-xlpxn\" (UID: \"dc44b87e-83bc-4d47-b262-120b31e7e65c\") " pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" Jul 06 19:26:33.695690 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.694012 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1e221582-b895-4724-8053-b72741d40132-etc-selinux\") pod \"aws-ebs-csi-driver-node-qndvc\" (UID: \"1e221582-b895-4724-8053-b72741d40132\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qndvc" Jul 06 19:26:33.695690 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.694037 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4e6171d1-c8d8-4800-8998-44acce38b440-multus-daemon-config\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.695690 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:33.694042 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:26:33.695690 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.694061 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/696eee70-dc27-4b63-b656-ab71bd2ebec5-dbus\") pod \"global-pull-secret-syncer-h7z7k\" (UID: \"696eee70-dc27-4b63-b656-ab71bd2ebec5\") " pod="kube-system/global-pull-secret-syncer-h7z7k" Jul 06 19:26:33.695690 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.694086 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/696eee70-dc27-4b63-b656-ab71bd2ebec5-original-pull-secret\") pod \"global-pull-secret-syncer-h7z7k\" (UID: \"696eee70-dc27-4b63-b656-ab71bd2ebec5\") " pod="kube-system/global-pull-secret-syncer-h7z7k" Jul 06 19:26:33.695690 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:33.694122 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2f29af8-1539-48c2-a77e-3b7691f5cea8-metrics-certs podName:c2f29af8-1539-48c2-a77e-3b7691f5cea8 nodeName:}" failed. No retries permitted until 2026-07-06 19:26:34.194087311 +0000 UTC m=+3.014176004 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c2f29af8-1539-48c2-a77e-3b7691f5cea8-metrics-certs") pod "network-metrics-daemon-ptml7" (UID: "c2f29af8-1539-48c2-a77e-3b7691f5cea8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:26:33.695690 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.694200 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bd815c1-1206-431e-9d76-ba36f79833ac-tmp-dir\") pod \"node-resolver-2xwfj\" (UID: \"9bd815c1-1206-431e-9d76-ba36f79833ac\") " pod="openshift-dns/node-resolver-2xwfj" Jul 06 19:26:33.695690 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.694243 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/189cc7dd-7ba3-4cfc-b51c-4a8ab328f621-cni-binary-copy\") pod \"multus-additional-cni-plugins-ttjjl\" (UID: \"189cc7dd-7ba3-4cfc-b51c-4a8ab328f621\") " pod="openshift-multus/multus-additional-cni-plugins-ttjjl" Jul 06 19:26:33.695690 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.694271 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/facd7b54-280f-4ee9-91df-3a0773f5dc0a-systemd-units\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.695690 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.694306 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/facd7b54-280f-4ee9-91df-3a0773f5dc0a-env-overrides\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.695690 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.694374 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/696eee70-dc27-4b63-b656-ab71bd2ebec5-kubelet-config\") pod \"global-pull-secret-syncer-h7z7k\" (UID: \"696eee70-dc27-4b63-b656-ab71bd2ebec5\") " pod="kube-system/global-pull-secret-syncer-h7z7k" Jul 06 19:26:33.695690 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.694433 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/556124af-4be3-4157-8b4b-70c1c1dfb6fb-agent-certs\") pod \"konnectivity-agent-pgh76\" (UID: \"556124af-4be3-4157-8b4b-70c1c1dfb6fb\") " pod="kube-system/konnectivity-agent-pgh76" Jul 06 19:26:33.695690 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.694747 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/189cc7dd-7ba3-4cfc-b51c-4a8ab328f621-cni-binary-copy\") pod \"multus-additional-cni-plugins-ttjjl\" (UID: \"189cc7dd-7ba3-4cfc-b51c-4a8ab328f621\") " pod="openshift-multus/multus-additional-cni-plugins-ttjjl" Jul 06 19:26:33.697235 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.697217 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc44b87e-83bc-4d47-b262-120b31e7e65c-tmp\") pod \"tuned-xlpxn\" (UID: \"dc44b87e-83bc-4d47-b262-120b31e7e65c\") " pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" Jul 06 19:26:33.697322 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.697275 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dc44b87e-83bc-4d47-b262-120b31e7e65c-etc-tuned\") pod \"tuned-xlpxn\" (UID: \"dc44b87e-83bc-4d47-b262-120b31e7e65c\") " pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" Jul 06 19:26:33.719562 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:33.719531 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:26:33.719562 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:33.719561 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:26:33.719781 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:33.719574 2575 projected.go:196] Error preparing data for projected volume kube-api-access-hk9j2 for pod openshift-network-diagnostics/network-check-target-x5xtb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:26:33.719781 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:33.719644 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/867a102b-d74b-4541-8505-4f22e0871dd1-kube-api-access-hk9j2 podName:867a102b-d74b-4541-8505-4f22e0871dd1 nodeName:}" failed. No retries permitted until 2026-07-06 19:26:34.219624626 +0000 UTC m=+3.039713320 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hk9j2" (UniqueName: "kubernetes.io/projected/867a102b-d74b-4541-8505-4f22e0871dd1-kube-api-access-hk9j2") pod "network-check-target-x5xtb" (UID: "867a102b-d74b-4541-8505-4f22e0871dd1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:26:33.722440 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.722414 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnwt8\" (UniqueName: \"kubernetes.io/projected/8561ed5c-290c-44b6-8490-8256078f657b-kube-api-access-rnwt8\") pod \"iptables-alerter-6px9t\" (UID: \"8561ed5c-290c-44b6-8490-8256078f657b\") " pod="openshift-network-operator/iptables-alerter-6px9t" Jul 06 19:26:33.722564 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.722439 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxlbh\" (UniqueName: \"kubernetes.io/projected/189cc7dd-7ba3-4cfc-b51c-4a8ab328f621-kube-api-access-zxlbh\") pod \"multus-additional-cni-plugins-ttjjl\" (UID: \"189cc7dd-7ba3-4cfc-b51c-4a8ab328f621\") " pod="openshift-multus/multus-additional-cni-plugins-ttjjl" Jul 06 19:26:33.722894 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.722868 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85pqf\" (UniqueName: \"kubernetes.io/projected/dc44b87e-83bc-4d47-b262-120b31e7e65c-kube-api-access-85pqf\") pod \"tuned-xlpxn\" (UID: \"dc44b87e-83bc-4d47-b262-120b31e7e65c\") " pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" Jul 06 19:26:33.723104 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.723080 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hmnl\" (UniqueName: \"kubernetes.io/projected/c2f29af8-1539-48c2-a77e-3b7691f5cea8-kube-api-access-5hmnl\") pod \"network-metrics-daemon-ptml7\" (UID: \"c2f29af8-1539-48c2-a77e-3b7691f5cea8\") " pod="openshift-multus/network-metrics-daemon-ptml7" Jul 06 19:26:33.793800 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.793742 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-23.ec2.internal" event={"ID":"b19b38d7a6af7f5567994a3fcadd5e98","Type":"ContainerStarted","Data":"65139535229a19fc662a4f25452cb33c4beae03fdda0274d4b407990076a5808"} Jul 06 19:26:33.794766 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.794733 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/facd7b54-280f-4ee9-91df-3a0773f5dc0a-host-cni-netd\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.794766 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.794766 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4e6171d1-c8d8-4800-8998-44acce38b440-host-run-multus-certs\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.794927 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.794793 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e6171d1-c8d8-4800-8998-44acce38b440-multus-cni-dir\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.794927 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.794833 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e6171d1-c8d8-4800-8998-44acce38b440-cni-binary-copy\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.794927 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.794856 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e6171d1-c8d8-4800-8998-44acce38b440-host-run-netns\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.794927 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.794876 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/facd7b54-280f-4ee9-91df-3a0773f5dc0a-host-cni-netd\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.794927 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.794883 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/556124af-4be3-4157-8b4b-70c1c1dfb6fb-konnectivity-ca\") pod \"konnectivity-agent-pgh76\" (UID: \"556124af-4be3-4157-8b4b-70c1c1dfb6fb\") " pod="kube-system/konnectivity-agent-pgh76" Jul 06 19:26:33.795166 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.794933 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1e221582-b895-4724-8053-b72741d40132-registration-dir\") pod \"aws-ebs-csi-driver-node-qndvc\" (UID: \"1e221582-b895-4724-8053-b72741d40132\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qndvc" Jul 06 19:26:33.795166 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.794955 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/facd7b54-280f-4ee9-91df-3a0773f5dc0a-host-slash\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.795166 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.794975 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/facd7b54-280f-4ee9-91df-3a0773f5dc0a-run-systemd\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.795166 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.794994 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4e6171d1-c8d8-4800-8998-44acce38b440-os-release\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.795166 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795021 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-23.ec2.internal" event={"ID":"42d465f5997f682b6201edde58cb1d3a","Type":"ContainerStarted","Data":"bfab5898f0c9d04da5a5d14de54a3557e95e23abcdc37891eb4cd5d48a97e258"} Jul 06 19:26:33.795166 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795032 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4e6171d1-c8d8-4800-8998-44acce38b440-host-run-k8s-cni-cncf-io\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.795166 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795055 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4e6171d1-c8d8-4800-8998-44acce38b440-host-var-lib-cni-multus\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.795166 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795066 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e6171d1-c8d8-4800-8998-44acce38b440-multus-cni-dir\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.795166 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795079 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1e221582-b895-4724-8053-b72741d40132-sys-fs\") pod \"aws-ebs-csi-driver-node-qndvc\" (UID: \"1e221582-b895-4724-8053-b72741d40132\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qndvc" Jul 06 19:26:33.795166 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795117 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4e6171d1-c8d8-4800-8998-44acce38b440-host-run-multus-certs\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.795166 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795126 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/facd7b54-280f-4ee9-91df-3a0773f5dc0a-run-systemd\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.795708 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795202 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1e221582-b895-4724-8053-b72741d40132-registration-dir\") pod \"aws-ebs-csi-driver-node-qndvc\" (UID: \"1e221582-b895-4724-8053-b72741d40132\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qndvc" Jul 06 19:26:33.795708 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795218 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/facd7b54-280f-4ee9-91df-3a0773f5dc0a-run-ovn\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.795708 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795245 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/facd7b54-280f-4ee9-91df-3a0773f5dc0a-host-slash\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.795708 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795257 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/facd7b54-280f-4ee9-91df-3a0773f5dc0a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.795708 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795282 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e6171d1-c8d8-4800-8998-44acce38b440-multus-conf-dir\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.795708 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795301 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e6171d1-c8d8-4800-8998-44acce38b440-host-run-netns\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.795708 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795303 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/863c8526-b822-48d4-8881-819331146578-host\") pod \"node-ca-q2tnp\" (UID: \"863c8526-b822-48d4-8881-819331146578\") " pod="openshift-image-registry/node-ca-q2tnp" Jul 06 19:26:33.795708 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795338 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/863c8526-b822-48d4-8881-819331146578-host\") pod \"node-ca-q2tnp\" (UID: \"863c8526-b822-48d4-8881-819331146578\") " pod="openshift-image-registry/node-ca-q2tnp" Jul 06 19:26:33.795708 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795337 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9bd815c1-1206-431e-9d76-ba36f79833ac-hosts-file\") pod \"node-resolver-2xwfj\" (UID: \"9bd815c1-1206-431e-9d76-ba36f79833ac\") " pod="openshift-dns/node-resolver-2xwfj" Jul 06 19:26:33.795708 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795386 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/facd7b54-280f-4ee9-91df-3a0773f5dc0a-host-kubelet\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.795708 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795398 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4e6171d1-c8d8-4800-8998-44acce38b440-os-release\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.795708 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795412 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/facd7b54-280f-4ee9-91df-3a0773f5dc0a-var-lib-openvswitch\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.795708 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795431 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4e6171d1-c8d8-4800-8998-44acce38b440-host-run-k8s-cni-cncf-io\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.795708 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795436 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e6171d1-c8d8-4800-8998-44acce38b440-cnibin\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.795708 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795460 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n4cf\" (UniqueName: \"kubernetes.io/projected/4e6171d1-c8d8-4800-8998-44acce38b440-kube-api-access-4n4cf\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.796416 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795467 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4e6171d1-c8d8-4800-8998-44acce38b440-host-var-lib-cni-multus\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.796416 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795497 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1e221582-b895-4724-8053-b72741d40132-device-dir\") pod \"aws-ebs-csi-driver-node-qndvc\" (UID: \"1e221582-b895-4724-8053-b72741d40132\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qndvc" Jul 06 19:26:33.796416 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795508 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e6171d1-c8d8-4800-8998-44acce38b440-multus-conf-dir\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.796416 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795520 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/facd7b54-280f-4ee9-91df-3a0773f5dc0a-host-cni-bin\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.796416 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795545 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1e221582-b895-4724-8053-b72741d40132-sys-fs\") pod \"aws-ebs-csi-driver-node-qndvc\" (UID: \"1e221582-b895-4724-8053-b72741d40132\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qndvc" Jul 06 19:26:33.796416 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795548 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/facd7b54-280f-4ee9-91df-3a0773f5dc0a-ovnkube-config\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.796416 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795553 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/facd7b54-280f-4ee9-91df-3a0773f5dc0a-run-ovn\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.796416 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795585 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4e6171d1-c8d8-4800-8998-44acce38b440-multus-socket-dir-parent\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.796416 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795611 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1e221582-b895-4724-8053-b72741d40132-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qndvc\" (UID: \"1e221582-b895-4724-8053-b72741d40132\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qndvc" Jul 06 19:26:33.796416 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795618 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e6171d1-c8d8-4800-8998-44acce38b440-cnibin\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.796416 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795659 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/facd7b54-280f-4ee9-91df-3a0773f5dc0a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.796416 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795667 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/facd7b54-280f-4ee9-91df-3a0773f5dc0a-host-kubelet\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.796416 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795700 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1e221582-b895-4724-8053-b72741d40132-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qndvc\" (UID: \"1e221582-b895-4724-8053-b72741d40132\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qndvc" Jul 06 19:26:33.796416 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795718 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e6171d1-c8d8-4800-8998-44acce38b440-cni-binary-copy\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.796416 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795460 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/556124af-4be3-4157-8b4b-70c1c1dfb6fb-konnectivity-ca\") pod \"konnectivity-agent-pgh76\" (UID: \"556124af-4be3-4157-8b4b-70c1c1dfb6fb\") " pod="kube-system/konnectivity-agent-pgh76" Jul 06 19:26:33.797112 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795751 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9bd815c1-1206-431e-9d76-ba36f79833ac-hosts-file\") pod \"node-resolver-2xwfj\" (UID: \"9bd815c1-1206-431e-9d76-ba36f79833ac\") " pod="openshift-dns/node-resolver-2xwfj" Jul 06 19:26:33.797112 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795768 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4s5q\" (UniqueName: \"kubernetes.io/projected/facd7b54-280f-4ee9-91df-3a0773f5dc0a-kube-api-access-l4s5q\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.797112 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795800 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqmpf\" (UniqueName: \"kubernetes.io/projected/863c8526-b822-48d4-8881-819331146578-kube-api-access-sqmpf\") pod \"node-ca-q2tnp\" (UID: \"863c8526-b822-48d4-8881-819331146578\") " pod="openshift-image-registry/node-ca-q2tnp" Jul 06 19:26:33.797112 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795807 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/facd7b54-280f-4ee9-91df-3a0773f5dc0a-var-lib-openvswitch\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.797112 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795854 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9txm\" (UniqueName: \"kubernetes.io/projected/1e221582-b895-4724-8053-b72741d40132-kube-api-access-f9txm\") pod \"aws-ebs-csi-driver-node-qndvc\" (UID: \"1e221582-b895-4724-8053-b72741d40132\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qndvc" Jul 06 19:26:33.797112 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795887 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/facd7b54-280f-4ee9-91df-3a0773f5dc0a-ovn-node-metrics-cert\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.797112 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795915 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e6171d1-c8d8-4800-8998-44acce38b440-host-var-lib-cni-bin\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.797112 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795938 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4e6171d1-c8d8-4800-8998-44acce38b440-hostroot\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.797112 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795962 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/facd7b54-280f-4ee9-91df-3a0773f5dc0a-ovnkube-script-lib\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.797112 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.795999 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/facd7b54-280f-4ee9-91df-3a0773f5dc0a-host-run-netns\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.797112 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.796023 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/facd7b54-280f-4ee9-91df-3a0773f5dc0a-etc-openvswitch\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.797112 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.796025 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1e221582-b895-4724-8053-b72741d40132-device-dir\") pod \"aws-ebs-csi-driver-node-qndvc\" (UID: \"1e221582-b895-4724-8053-b72741d40132\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qndvc" Jul 06 19:26:33.797112 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.796044 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/facd7b54-280f-4ee9-91df-3a0773f5dc0a-node-log\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.797112 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.796070 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/facd7b54-280f-4ee9-91df-3a0773f5dc0a-host-run-ovn-kubernetes\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.797733 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.796101 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e6171d1-c8d8-4800-8998-44acce38b440-system-cni-dir\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.797733 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.796128 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e6171d1-c8d8-4800-8998-44acce38b440-etc-kubernetes\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.797733 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.796132 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4e6171d1-c8d8-4800-8998-44acce38b440-multus-socket-dir-parent\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.797733 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.796144 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/facd7b54-280f-4ee9-91df-3a0773f5dc0a-host-cni-bin\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.797733 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.796153 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/863c8526-b822-48d4-8881-819331146578-serviceca\") pod \"node-ca-q2tnp\" (UID: \"863c8526-b822-48d4-8881-819331146578\") " pod="openshift-image-registry/node-ca-q2tnp" Jul 06 19:26:33.797733 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.796205 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/facd7b54-280f-4ee9-91df-3a0773f5dc0a-host-run-ovn-kubernetes\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.797733 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.796215 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1e221582-b895-4724-8053-b72741d40132-etc-selinux\") pod \"aws-ebs-csi-driver-node-qndvc\" (UID: \"1e221582-b895-4724-8053-b72741d40132\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qndvc" Jul 06 19:26:33.797733 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.796246 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4e6171d1-c8d8-4800-8998-44acce38b440-multus-daemon-config\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.797733 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.796251 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4e6171d1-c8d8-4800-8998-44acce38b440-hostroot\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.797733 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.796271 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/696eee70-dc27-4b63-b656-ab71bd2ebec5-dbus\") pod \"global-pull-secret-syncer-h7z7k\" (UID: \"696eee70-dc27-4b63-b656-ab71bd2ebec5\") " pod="kube-system/global-pull-secret-syncer-h7z7k" Jul 06 19:26:33.797733 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.796297 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/696eee70-dc27-4b63-b656-ab71bd2ebec5-original-pull-secret\") pod \"global-pull-secret-syncer-h7z7k\" (UID: \"696eee70-dc27-4b63-b656-ab71bd2ebec5\") " pod="kube-system/global-pull-secret-syncer-h7z7k" Jul 06 19:26:33.797733 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.796321 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bd815c1-1206-431e-9d76-ba36f79833ac-tmp-dir\") pod \"node-resolver-2xwfj\" (UID: \"9bd815c1-1206-431e-9d76-ba36f79833ac\") " pod="openshift-dns/node-resolver-2xwfj" Jul 06 19:26:33.797733 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.796332 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/facd7b54-280f-4ee9-91df-3a0773f5dc0a-etc-openvswitch\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.797733 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.796344 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/facd7b54-280f-4ee9-91df-3a0773f5dc0a-systemd-units\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.797733 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.796370 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/facd7b54-280f-4ee9-91df-3a0773f5dc0a-host-run-netns\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.798394 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.796371 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/facd7b54-280f-4ee9-91df-3a0773f5dc0a-env-overrides\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.798394 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.796424 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/696eee70-dc27-4b63-b656-ab71bd2ebec5-kubelet-config\") pod \"global-pull-secret-syncer-h7z7k\" (UID: \"696eee70-dc27-4b63-b656-ab71bd2ebec5\") " pod="kube-system/global-pull-secret-syncer-h7z7k" Jul 06 19:26:33.798394 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.796449 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/556124af-4be3-4157-8b4b-70c1c1dfb6fb-agent-certs\") pod \"konnectivity-agent-pgh76\" (UID: \"556124af-4be3-4157-8b4b-70c1c1dfb6fb\") " pod="kube-system/konnectivity-agent-pgh76" Jul 06 19:26:33.798394 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.796476 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/facd7b54-280f-4ee9-91df-3a0773f5dc0a-log-socket\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.798394 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.796523 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e6171d1-c8d8-4800-8998-44acce38b440-host-var-lib-kubelet\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.798394 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.796557 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht2bf\" (UniqueName: \"kubernetes.io/projected/9bd815c1-1206-431e-9d76-ba36f79833ac-kube-api-access-ht2bf\") pod \"node-resolver-2xwfj\" (UID: \"9bd815c1-1206-431e-9d76-ba36f79833ac\") " pod="openshift-dns/node-resolver-2xwfj" Jul 06 19:26:33.798394 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.796594 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1e221582-b895-4724-8053-b72741d40132-socket-dir\") pod \"aws-ebs-csi-driver-node-qndvc\" (UID: \"1e221582-b895-4724-8053-b72741d40132\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qndvc" Jul 06 19:26:33.798394 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.796650 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/facd7b54-280f-4ee9-91df-3a0773f5dc0a-run-openvswitch\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.798394 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.797121 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/facd7b54-280f-4ee9-91df-3a0773f5dc0a-ovnkube-script-lib\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.798394 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.797166 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/facd7b54-280f-4ee9-91df-3a0773f5dc0a-env-overrides\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.798394 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.796073 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e6171d1-c8d8-4800-8998-44acce38b440-host-var-lib-cni-bin\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.798394 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.796073 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/facd7b54-280f-4ee9-91df-3a0773f5dc0a-ovnkube-config\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.798394 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:33.797477 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:26:33.798394 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.797568 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e6171d1-c8d8-4800-8998-44acce38b440-system-cni-dir\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.798979 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:33.797837 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/696eee70-dc27-4b63-b656-ab71bd2ebec5-original-pull-secret podName:696eee70-dc27-4b63-b656-ab71bd2ebec5 nodeName:}" failed. No retries permitted until 2026-07-06 19:26:34.297810455 +0000 UTC m=+3.117899144 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/696eee70-dc27-4b63-b656-ab71bd2ebec5-original-pull-secret") pod "global-pull-secret-syncer-h7z7k" (UID: "696eee70-dc27-4b63-b656-ab71bd2ebec5") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:26:33.798979 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.797862 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bd815c1-1206-431e-9d76-ba36f79833ac-tmp-dir\") pod \"node-resolver-2xwfj\" (UID: \"9bd815c1-1206-431e-9d76-ba36f79833ac\") " pod="openshift-dns/node-resolver-2xwfj" Jul 06 19:26:33.798979 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.797907 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e6171d1-c8d8-4800-8998-44acce38b440-etc-kubernetes\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.798979 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.797915 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/facd7b54-280f-4ee9-91df-3a0773f5dc0a-run-openvswitch\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.798979 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.797965 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e6171d1-c8d8-4800-8998-44acce38b440-host-var-lib-kubelet\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.798979 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.798007 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1e221582-b895-4724-8053-b72741d40132-etc-selinux\") pod \"aws-ebs-csi-driver-node-qndvc\" (UID: \"1e221582-b895-4724-8053-b72741d40132\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qndvc" Jul 06 19:26:33.798979 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.798012 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/facd7b54-280f-4ee9-91df-3a0773f5dc0a-log-socket\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.798979 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.798111 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1e221582-b895-4724-8053-b72741d40132-socket-dir\") pod \"aws-ebs-csi-driver-node-qndvc\" (UID: \"1e221582-b895-4724-8053-b72741d40132\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qndvc" Jul 06 19:26:33.798979 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.798176 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/facd7b54-280f-4ee9-91df-3a0773f5dc0a-node-log\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.798979 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.798253 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/facd7b54-280f-4ee9-91df-3a0773f5dc0a-systemd-units\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.798979 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.798314 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/696eee70-dc27-4b63-b656-ab71bd2ebec5-dbus\") pod \"global-pull-secret-syncer-h7z7k\" (UID: \"696eee70-dc27-4b63-b656-ab71bd2ebec5\") " pod="kube-system/global-pull-secret-syncer-h7z7k" Jul 06 19:26:33.798979 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.798433 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/696eee70-dc27-4b63-b656-ab71bd2ebec5-kubelet-config\") pod \"global-pull-secret-syncer-h7z7k\" (UID: \"696eee70-dc27-4b63-b656-ab71bd2ebec5\") " pod="kube-system/global-pull-secret-syncer-h7z7k" Jul 06 19:26:33.798979 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.798520 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/863c8526-b822-48d4-8881-819331146578-serviceca\") pod \"node-ca-q2tnp\" (UID: \"863c8526-b822-48d4-8881-819331146578\") " pod="openshift-image-registry/node-ca-q2tnp" Jul 06 19:26:33.798979 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.798651 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4e6171d1-c8d8-4800-8998-44acce38b440-multus-daemon-config\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.802635 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.802582 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/facd7b54-280f-4ee9-91df-3a0773f5dc0a-ovn-node-metrics-cert\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.803046 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.803002 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/556124af-4be3-4157-8b4b-70c1c1dfb6fb-agent-certs\") pod \"konnectivity-agent-pgh76\" (UID: \"556124af-4be3-4157-8b4b-70c1c1dfb6fb\") " pod="kube-system/konnectivity-agent-pgh76" Jul 06 19:26:33.815319 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.815253 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n4cf\" (UniqueName: \"kubernetes.io/projected/4e6171d1-c8d8-4800-8998-44acce38b440-kube-api-access-4n4cf\") pod \"multus-dw4z9\" (UID: \"4e6171d1-c8d8-4800-8998-44acce38b440\") " pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.818110 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.818083 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqmpf\" (UniqueName: \"kubernetes.io/projected/863c8526-b822-48d4-8881-819331146578-kube-api-access-sqmpf\") pod \"node-ca-q2tnp\" (UID: \"863c8526-b822-48d4-8881-819331146578\") " pod="openshift-image-registry/node-ca-q2tnp" Jul 06 19:26:33.818271 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.818224 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht2bf\" (UniqueName: \"kubernetes.io/projected/9bd815c1-1206-431e-9d76-ba36f79833ac-kube-api-access-ht2bf\") pod \"node-resolver-2xwfj\" (UID: \"9bd815c1-1206-431e-9d76-ba36f79833ac\") " pod="openshift-dns/node-resolver-2xwfj" Jul 06 19:26:33.818759 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.818721 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9txm\" (UniqueName: \"kubernetes.io/projected/1e221582-b895-4724-8053-b72741d40132-kube-api-access-f9txm\") pod \"aws-ebs-csi-driver-node-qndvc\" (UID: \"1e221582-b895-4724-8053-b72741d40132\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qndvc" Jul 06 19:26:33.818857 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.818771 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4s5q\" (UniqueName: \"kubernetes.io/projected/facd7b54-280f-4ee9-91df-3a0773f5dc0a-kube-api-access-l4s5q\") pod \"ovnkube-node-965cw\" (UID: \"facd7b54-280f-4ee9-91df-3a0773f5dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.888741 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.888715 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" Jul 06 19:26:33.896398 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.896379 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ttjjl" Jul 06 19:26:33.907061 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.907029 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6px9t" Jul 06 19:26:33.916469 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.916432 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:33.924387 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.924363 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dw4z9" Jul 06 19:26:33.931207 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.931170 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q2tnp" Jul 06 19:26:33.941022 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.941002 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2xwfj" Jul 06 19:26:33.947174 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.947156 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pgh76" Jul 06 19:26:33.958797 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:33.958777 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qndvc" Jul 06 19:26:34.200194 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:34.200102 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2f29af8-1539-48c2-a77e-3b7691f5cea8-metrics-certs\") pod \"network-metrics-daemon-ptml7\" (UID: \"c2f29af8-1539-48c2-a77e-3b7691f5cea8\") " pod="openshift-multus/network-metrics-daemon-ptml7" Jul 06 19:26:34.200335 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:34.200242 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:26:34.200335 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:34.200295 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2f29af8-1539-48c2-a77e-3b7691f5cea8-metrics-certs podName:c2f29af8-1539-48c2-a77e-3b7691f5cea8 nodeName:}" failed. No retries permitted until 2026-07-06 19:26:35.200281781 +0000 UTC m=+4.020370471 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c2f29af8-1539-48c2-a77e-3b7691f5cea8-metrics-certs") pod "network-metrics-daemon-ptml7" (UID: "c2f29af8-1539-48c2-a77e-3b7691f5cea8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:26:34.300689 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:34.300652 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk9j2\" (UniqueName: \"kubernetes.io/projected/867a102b-d74b-4541-8505-4f22e0871dd1-kube-api-access-hk9j2\") pod \"network-check-target-x5xtb\" (UID: \"867a102b-d74b-4541-8505-4f22e0871dd1\") " pod="openshift-network-diagnostics/network-check-target-x5xtb" Jul 06 19:26:34.300883 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:34.300725 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/696eee70-dc27-4b63-b656-ab71bd2ebec5-original-pull-secret\") pod \"global-pull-secret-syncer-h7z7k\" (UID: \"696eee70-dc27-4b63-b656-ab71bd2ebec5\") " pod="kube-system/global-pull-secret-syncer-h7z7k" Jul 06 19:26:34.300883 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:34.300837 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:26:34.300883 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:34.300842 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:26:34.300883 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:34.300868 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:26:34.300883 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:34.300882 2575 projected.go:196] Error preparing data for projected volume kube-api-access-hk9j2 for pod openshift-network-diagnostics/network-check-target-x5xtb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:26:34.301120 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:34.300904 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/696eee70-dc27-4b63-b656-ab71bd2ebec5-original-pull-secret podName:696eee70-dc27-4b63-b656-ab71bd2ebec5 nodeName:}" failed. No retries permitted until 2026-07-06 19:26:35.300885818 +0000 UTC m=+4.120974521 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/696eee70-dc27-4b63-b656-ab71bd2ebec5-original-pull-secret") pod "global-pull-secret-syncer-h7z7k" (UID: "696eee70-dc27-4b63-b656-ab71bd2ebec5") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:26:34.301120 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:34.300925 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/867a102b-d74b-4541-8505-4f22e0871dd1-kube-api-access-hk9j2 podName:867a102b-d74b-4541-8505-4f22e0871dd1 nodeName:}" failed. No retries permitted until 2026-07-06 19:26:35.300912994 +0000 UTC m=+4.121001684 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hk9j2" (UniqueName: "kubernetes.io/projected/867a102b-d74b-4541-8505-4f22e0871dd1-kube-api-access-hk9j2") pod "network-check-target-x5xtb" (UID: "867a102b-d74b-4541-8505-4f22e0871dd1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:26:34.581877 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:34.581803 2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:26:34.640609 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:34.640541 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 19:21:33 +0000 UTC" deadline="2028-02-19 08:17:22.476449278 +0000 UTC" Jul 06 19:26:34.640609 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:34.640594 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14220h50m47.835860298s" Jul 06 19:26:34.698952 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:34.698927 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfacd7b54_280f_4ee9_91df_3a0773f5dc0a.slice/crio-036ec5acc63e7af42af21a3f572abe15a4a90bddbbb939d5247f459d60171d80 WatchSource:0}: Error finding container 036ec5acc63e7af42af21a3f572abe15a4a90bddbbb939d5247f459d60171d80: Status 404 returned error can't find the container with id 036ec5acc63e7af42af21a3f572abe15a4a90bddbbb939d5247f459d60171d80 Jul 06 19:26:34.700740 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:34.700717 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e221582_b895_4724_8053_b72741d40132.slice/crio-646211f3fd7e6791b11924581d4901191c5c203eea600c6cddc85037024fc332 WatchSource:0}: Error finding container 646211f3fd7e6791b11924581d4901191c5c203eea600c6cddc85037024fc332: Status 404 returned error can't find the container with id 646211f3fd7e6791b11924581d4901191c5c203eea600c6cddc85037024fc332 Jul 06 19:26:34.701716 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:34.701693 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8561ed5c_290c_44b6_8490_8256078f657b.slice/crio-94f0077415032165cd3ec658bf442e7f5e451399ec6a9d5ac87125ec587ac9eb WatchSource:0}: Error finding container 94f0077415032165cd3ec658bf442e7f5e451399ec6a9d5ac87125ec587ac9eb: Status 404 returned error can't find the container with id 94f0077415032165cd3ec658bf442e7f5e451399ec6a9d5ac87125ec587ac9eb Jul 06 19:26:34.703132 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:34.703101 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e6171d1_c8d8_4800_8998_44acce38b440.slice/crio-9316d9629f344ae4061153d4fcc78d0ec60c889577a1b399621d470d56ad1aab WatchSource:0}: Error finding container 9316d9629f344ae4061153d4fcc78d0ec60c889577a1b399621d470d56ad1aab: Status 404 returned error can't find the container with id 9316d9629f344ae4061153d4fcc78d0ec60c889577a1b399621d470d56ad1aab Jul 06 19:26:34.706598 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:34.706273 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bd815c1_1206_431e_9d76_ba36f79833ac.slice/crio-dc7a1aca3274859c5e987296bf64057abe0b61bada497884a44965d5d3da2aee WatchSource:0}: Error finding container dc7a1aca3274859c5e987296bf64057abe0b61bada497884a44965d5d3da2aee: Status 404 returned error can't find the container with id dc7a1aca3274859c5e987296bf64057abe0b61bada497884a44965d5d3da2aee Jul 06 19:26:34.708772 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:34.708734 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod556124af_4be3_4157_8b4b_70c1c1dfb6fb.slice/crio-028a0805a0fec9ac25d3ff5e3d18a1d28312b0760430dace76f8684a24bd7fa8 WatchSource:0}: Error finding container 028a0805a0fec9ac25d3ff5e3d18a1d28312b0760430dace76f8684a24bd7fa8: Status 404 returned error can't find the container with id 028a0805a0fec9ac25d3ff5e3d18a1d28312b0760430dace76f8684a24bd7fa8 Jul 06 19:26:34.709323 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:26:34.709208 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod863c8526_b822_48d4_8881_819331146578.slice/crio-b73440e78d17f21174edc56e0d63e3b781a2a7dc5e12116de47a687eb7e25a64 WatchSource:0}: Error finding container b73440e78d17f21174edc56e0d63e3b781a2a7dc5e12116de47a687eb7e25a64: Status 404 returned error can't find the container with id b73440e78d17f21174edc56e0d63e3b781a2a7dc5e12116de47a687eb7e25a64 Jul 06 19:26:34.788847 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:34.788718 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7z7k" Jul 06 19:26:34.788947 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:34.788718 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ptml7" Jul 06 19:26:34.789004 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:34.788983 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7z7k" podUID="696eee70-dc27-4b63-b656-ab71bd2ebec5" Jul 06 19:26:34.789095 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:34.789071 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ptml7" podUID="c2f29af8-1539-48c2-a77e-3b7691f5cea8" Jul 06 19:26:34.798436 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:34.798412 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-965cw" event={"ID":"facd7b54-280f-4ee9-91df-3a0773f5dc0a","Type":"ContainerStarted","Data":"036ec5acc63e7af42af21a3f572abe15a4a90bddbbb939d5247f459d60171d80"} Jul 06 19:26:34.801053 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:34.801034 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-23.ec2.internal" event={"ID":"42d465f5997f682b6201edde58cb1d3a","Type":"ContainerStarted","Data":"92cf85021ed5712251eef6ba1b2bed73bc19978e0b5511154a28730c2f143524"} Jul 06 19:26:34.802341 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:34.802322 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q2tnp" event={"ID":"863c8526-b822-48d4-8881-819331146578","Type":"ContainerStarted","Data":"b73440e78d17f21174edc56e0d63e3b781a2a7dc5e12116de47a687eb7e25a64"} Jul 06 19:26:34.803386 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:34.803368 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pgh76" event={"ID":"556124af-4be3-4157-8b4b-70c1c1dfb6fb","Type":"ContainerStarted","Data":"028a0805a0fec9ac25d3ff5e3d18a1d28312b0760430dace76f8684a24bd7fa8"} Jul 06 19:26:34.804325 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:34.804305 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttjjl" event={"ID":"189cc7dd-7ba3-4cfc-b51c-4a8ab328f621","Type":"ContainerStarted","Data":"5a12d47c68110c9ac2e304394f3fc996c67e760e9b09d5c27e64519facbd1458"} Jul 06 19:26:34.805420 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:34.805398 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2xwfj" event={"ID":"9bd815c1-1206-431e-9d76-ba36f79833ac","Type":"ContainerStarted","Data":"dc7a1aca3274859c5e987296bf64057abe0b61bada497884a44965d5d3da2aee"} Jul 06 19:26:34.806621 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:34.806598 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dw4z9" event={"ID":"4e6171d1-c8d8-4800-8998-44acce38b440","Type":"ContainerStarted","Data":"9316d9629f344ae4061153d4fcc78d0ec60c889577a1b399621d470d56ad1aab"} Jul 06 19:26:34.807621 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:34.807599 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" event={"ID":"dc44b87e-83bc-4d47-b262-120b31e7e65c","Type":"ContainerStarted","Data":"2d7899de3ff004fdae3db5549e429dc509a8623a3b4326598cd8ad1a426af320"} Jul 06 19:26:34.808856 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:34.808823 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6px9t" event={"ID":"8561ed5c-290c-44b6-8490-8256078f657b","Type":"ContainerStarted","Data":"94f0077415032165cd3ec658bf442e7f5e451399ec6a9d5ac87125ec587ac9eb"} Jul 06 19:26:34.809873 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:34.809853 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qndvc" event={"ID":"1e221582-b895-4724-8053-b72741d40132","Type":"ContainerStarted","Data":"646211f3fd7e6791b11924581d4901191c5c203eea600c6cddc85037024fc332"} Jul 06 19:26:35.205932 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:35.205843 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2f29af8-1539-48c2-a77e-3b7691f5cea8-metrics-certs\") pod \"network-metrics-daemon-ptml7\" (UID: \"c2f29af8-1539-48c2-a77e-3b7691f5cea8\") " pod="openshift-multus/network-metrics-daemon-ptml7" Jul 06 19:26:35.206143 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:35.206045 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:26:35.206143 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:35.206111 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2f29af8-1539-48c2-a77e-3b7691f5cea8-metrics-certs podName:c2f29af8-1539-48c2-a77e-3b7691f5cea8 nodeName:}" failed. No retries permitted until 2026-07-06 19:26:37.206091367 +0000 UTC m=+6.026180061 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c2f29af8-1539-48c2-a77e-3b7691f5cea8-metrics-certs") pod "network-metrics-daemon-ptml7" (UID: "c2f29af8-1539-48c2-a77e-3b7691f5cea8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:26:35.306966 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:35.306914 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/696eee70-dc27-4b63-b656-ab71bd2ebec5-original-pull-secret\") pod \"global-pull-secret-syncer-h7z7k\" (UID: \"696eee70-dc27-4b63-b656-ab71bd2ebec5\") " pod="kube-system/global-pull-secret-syncer-h7z7k" Jul 06 19:26:35.307155 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:35.306997 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk9j2\" (UniqueName: \"kubernetes.io/projected/867a102b-d74b-4541-8505-4f22e0871dd1-kube-api-access-hk9j2\") pod \"network-check-target-x5xtb\" (UID: \"867a102b-d74b-4541-8505-4f22e0871dd1\") " pod="openshift-network-diagnostics/network-check-target-x5xtb" Jul 06 19:26:35.307281 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:35.307153 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:26:35.307281 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:35.307173 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:26:35.307281 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:35.307200 2575 projected.go:196] Error preparing data for projected volume kube-api-access-hk9j2 for pod openshift-network-diagnostics/network-check-target-x5xtb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:26:35.307281 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:35.307263 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/867a102b-d74b-4541-8505-4f22e0871dd1-kube-api-access-hk9j2 podName:867a102b-d74b-4541-8505-4f22e0871dd1 nodeName:}" failed. No retries permitted until 2026-07-06 19:26:37.307242612 +0000 UTC m=+6.127331323 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hk9j2" (UniqueName: "kubernetes.io/projected/867a102b-d74b-4541-8505-4f22e0871dd1-kube-api-access-hk9j2") pod "network-check-target-x5xtb" (UID: "867a102b-d74b-4541-8505-4f22e0871dd1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:26:35.307509 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:35.307343 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:26:35.307509 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:35.307379 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/696eee70-dc27-4b63-b656-ab71bd2ebec5-original-pull-secret podName:696eee70-dc27-4b63-b656-ab71bd2ebec5 nodeName:}" failed. No retries permitted until 2026-07-06 19:26:37.307367156 +0000 UTC m=+6.127455849 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/696eee70-dc27-4b63-b656-ab71bd2ebec5-original-pull-secret") pod "global-pull-secret-syncer-h7z7k" (UID: "696eee70-dc27-4b63-b656-ab71bd2ebec5") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:26:35.641104 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:35.640979 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 19:21:33 +0000 UTC" deadline="2028-01-04 11:44:53.437319186 +0000 UTC" Jul 06 19:26:35.641104 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:35.641066 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13120h18m17.796257808s" Jul 06 19:26:35.791050 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:35.790567 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5xtb" Jul 06 19:26:35.791050 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:35.790694 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5xtb" podUID="867a102b-d74b-4541-8505-4f22e0871dd1" Jul 06 19:26:35.828368 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:35.827134 2575 generic.go:356] "Generic (PLEG): container finished" podID="b19b38d7a6af7f5567994a3fcadd5e98" containerID="d24d1ce490c35eae40eb32735e9260c8c095a182dfa016a5f657035b399c7623" exitCode=0 Jul 06 19:26:35.828368 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:35.828113 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-23.ec2.internal" event={"ID":"b19b38d7a6af7f5567994a3fcadd5e98","Type":"ContainerDied","Data":"d24d1ce490c35eae40eb32735e9260c8c095a182dfa016a5f657035b399c7623"} Jul 06 19:26:35.846820 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:35.846658 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-23.ec2.internal" podStartSLOduration=2.846636909 podStartE2EDuration="2.846636909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:26:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:26:34.817241488 +0000 UTC m=+3.637330199" watchObservedRunningTime="2026-07-06 19:26:35.846636909 +0000 UTC m=+4.666725622" Jul 06 19:26:36.788727 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:36.788681 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7z7k" Jul 06 19:26:36.788928 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:36.788834 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7z7k" podUID="696eee70-dc27-4b63-b656-ab71bd2ebec5" Jul 06 19:26:36.789343 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:36.789311 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ptml7" Jul 06 19:26:36.789487 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:36.789454 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ptml7" podUID="c2f29af8-1539-48c2-a77e-3b7691f5cea8" Jul 06 19:26:36.881044 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:36.880963 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-23.ec2.internal" event={"ID":"b19b38d7a6af7f5567994a3fcadd5e98","Type":"ContainerStarted","Data":"c6af513b37d66277165ec8fbeb52f64d29c2e0aa553d78adb0bdce9504a367ae"} Jul 06 19:26:37.224983 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:37.224904 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2f29af8-1539-48c2-a77e-3b7691f5cea8-metrics-certs\") pod \"network-metrics-daemon-ptml7\" (UID: \"c2f29af8-1539-48c2-a77e-3b7691f5cea8\") " pod="openshift-multus/network-metrics-daemon-ptml7" Jul 06 19:26:37.225140 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:37.225057 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:26:37.225140 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:37.225120 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2f29af8-1539-48c2-a77e-3b7691f5cea8-metrics-certs podName:c2f29af8-1539-48c2-a77e-3b7691f5cea8 nodeName:}" failed. No retries permitted until 2026-07-06 19:26:41.225103269 +0000 UTC m=+10.045191973 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c2f29af8-1539-48c2-a77e-3b7691f5cea8-metrics-certs") pod "network-metrics-daemon-ptml7" (UID: "c2f29af8-1539-48c2-a77e-3b7691f5cea8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:26:37.325923 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:37.325874 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk9j2\" (UniqueName: \"kubernetes.io/projected/867a102b-d74b-4541-8505-4f22e0871dd1-kube-api-access-hk9j2\") pod \"network-check-target-x5xtb\" (UID: \"867a102b-d74b-4541-8505-4f22e0871dd1\") " pod="openshift-network-diagnostics/network-check-target-x5xtb" Jul 06 19:26:37.326104 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:37.325959 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/696eee70-dc27-4b63-b656-ab71bd2ebec5-original-pull-secret\") pod \"global-pull-secret-syncer-h7z7k\" (UID: \"696eee70-dc27-4b63-b656-ab71bd2ebec5\") " pod="kube-system/global-pull-secret-syncer-h7z7k" Jul 06 19:26:37.326171 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:37.326113 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:26:37.326255 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:37.326174 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/696eee70-dc27-4b63-b656-ab71bd2ebec5-original-pull-secret podName:696eee70-dc27-4b63-b656-ab71bd2ebec5 nodeName:}" failed. No retries permitted until 2026-07-06 19:26:41.326156769 +0000 UTC m=+10.146245469 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/696eee70-dc27-4b63-b656-ab71bd2ebec5-original-pull-secret") pod "global-pull-secret-syncer-h7z7k" (UID: "696eee70-dc27-4b63-b656-ab71bd2ebec5") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:26:37.326323 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:37.326281 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:26:37.326323 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:37.326296 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:26:37.326323 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:37.326309 2575 projected.go:196] Error preparing data for projected volume kube-api-access-hk9j2 for pod openshift-network-diagnostics/network-check-target-x5xtb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:26:37.326464 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:37.326345 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/867a102b-d74b-4541-8505-4f22e0871dd1-kube-api-access-hk9j2 podName:867a102b-d74b-4541-8505-4f22e0871dd1 nodeName:}" failed. No retries permitted until 2026-07-06 19:26:41.326333852 +0000 UTC m=+10.146422545 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hk9j2" (UniqueName: "kubernetes.io/projected/867a102b-d74b-4541-8505-4f22e0871dd1-kube-api-access-hk9j2") pod "network-check-target-x5xtb" (UID: "867a102b-d74b-4541-8505-4f22e0871dd1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:26:37.788124 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:37.788046 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5xtb" Jul 06 19:26:37.788367 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:37.788175 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5xtb" podUID="867a102b-d74b-4541-8505-4f22e0871dd1" Jul 06 19:26:38.788647 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:38.788608 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7z7k" Jul 06 19:26:38.789096 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:38.788657 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ptml7" Jul 06 19:26:38.789096 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:38.788745 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7z7k" podUID="696eee70-dc27-4b63-b656-ab71bd2ebec5" Jul 06 19:26:38.789096 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:38.788922 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ptml7" podUID="c2f29af8-1539-48c2-a77e-3b7691f5cea8" Jul 06 19:26:39.789205 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:39.789098 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5xtb" Jul 06 19:26:39.789660 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:39.789248 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5xtb" podUID="867a102b-d74b-4541-8505-4f22e0871dd1" Jul 06 19:26:40.789095 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:40.789062 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7z7k" Jul 06 19:26:40.789304 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:40.789065 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ptml7" Jul 06 19:26:40.789304 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:40.789218 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7z7k" podUID="696eee70-dc27-4b63-b656-ab71bd2ebec5" Jul 06 19:26:40.789696 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:40.789316 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ptml7" podUID="c2f29af8-1539-48c2-a77e-3b7691f5cea8" Jul 06 19:26:41.258668 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:41.258596 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2f29af8-1539-48c2-a77e-3b7691f5cea8-metrics-certs\") pod \"network-metrics-daemon-ptml7\" (UID: \"c2f29af8-1539-48c2-a77e-3b7691f5cea8\") " pod="openshift-multus/network-metrics-daemon-ptml7" Jul 06 19:26:41.258841 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:41.258763 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:26:41.258841 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:41.258820 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2f29af8-1539-48c2-a77e-3b7691f5cea8-metrics-certs podName:c2f29af8-1539-48c2-a77e-3b7691f5cea8 nodeName:}" failed. No retries permitted until 2026-07-06 19:26:49.258803372 +0000 UTC m=+18.078892067 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c2f29af8-1539-48c2-a77e-3b7691f5cea8-metrics-certs") pod "network-metrics-daemon-ptml7" (UID: "c2f29af8-1539-48c2-a77e-3b7691f5cea8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:26:41.359501 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:41.359468 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/696eee70-dc27-4b63-b656-ab71bd2ebec5-original-pull-secret\") pod \"global-pull-secret-syncer-h7z7k\" (UID: \"696eee70-dc27-4b63-b656-ab71bd2ebec5\") " pod="kube-system/global-pull-secret-syncer-h7z7k" Jul 06 19:26:41.359676 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:41.359540 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk9j2\" (UniqueName: \"kubernetes.io/projected/867a102b-d74b-4541-8505-4f22e0871dd1-kube-api-access-hk9j2\") pod \"network-check-target-x5xtb\" (UID: \"867a102b-d74b-4541-8505-4f22e0871dd1\") " pod="openshift-network-diagnostics/network-check-target-x5xtb" Jul 06 19:26:41.359676 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:41.359630 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:26:41.359676 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:41.359668 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:26:41.359818 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:41.359686 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:26:41.359818 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:41.359695 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/696eee70-dc27-4b63-b656-ab71bd2ebec5-original-pull-secret podName:696eee70-dc27-4b63-b656-ab71bd2ebec5 nodeName:}" failed. No retries permitted until 2026-07-06 19:26:49.359675625 +0000 UTC m=+18.179764314 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/696eee70-dc27-4b63-b656-ab71bd2ebec5-original-pull-secret") pod "global-pull-secret-syncer-h7z7k" (UID: "696eee70-dc27-4b63-b656-ab71bd2ebec5") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:26:41.359818 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:41.359701 2575 projected.go:196] Error preparing data for projected volume kube-api-access-hk9j2 for pod openshift-network-diagnostics/network-check-target-x5xtb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:26:41.359818 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:41.359748 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/867a102b-d74b-4541-8505-4f22e0871dd1-kube-api-access-hk9j2 podName:867a102b-d74b-4541-8505-4f22e0871dd1 nodeName:}" failed. No retries permitted until 2026-07-06 19:26:49.359733592 +0000 UTC m=+18.179822285 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hk9j2" (UniqueName: "kubernetes.io/projected/867a102b-d74b-4541-8505-4f22e0871dd1-kube-api-access-hk9j2") pod "network-check-target-x5xtb" (UID: "867a102b-d74b-4541-8505-4f22e0871dd1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:26:41.789047 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:41.789013 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5xtb" Jul 06 19:26:41.789305 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:41.789151 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5xtb" podUID="867a102b-d74b-4541-8505-4f22e0871dd1" Jul 06 19:26:42.788450 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:42.788374 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ptml7" Jul 06 19:26:42.788450 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:42.788390 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7z7k" Jul 06 19:26:42.788671 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:42.788514 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7z7k" podUID="696eee70-dc27-4b63-b656-ab71bd2ebec5" Jul 06 19:26:42.788833 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:42.788784 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ptml7" podUID="c2f29af8-1539-48c2-a77e-3b7691f5cea8" Jul 06 19:26:43.788771 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:43.788740 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5xtb" Jul 06 19:26:43.789161 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:43.788869 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5xtb" podUID="867a102b-d74b-4541-8505-4f22e0871dd1" Jul 06 19:26:44.788934 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:44.788898 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ptml7" Jul 06 19:26:44.789479 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:44.789032 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ptml7" podUID="c2f29af8-1539-48c2-a77e-3b7691f5cea8" Jul 06 19:26:44.789479 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:44.789093 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7z7k" Jul 06 19:26:44.789479 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:44.789226 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7z7k" podUID="696eee70-dc27-4b63-b656-ab71bd2ebec5" Jul 06 19:26:45.788883 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:45.788853 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5xtb" Jul 06 19:26:45.789043 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:45.788959 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5xtb" podUID="867a102b-d74b-4541-8505-4f22e0871dd1" Jul 06 19:26:46.788978 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:46.788944 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7z7k" Jul 06 19:26:46.789140 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:46.789060 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7z7k" podUID="696eee70-dc27-4b63-b656-ab71bd2ebec5" Jul 06 19:26:46.789140 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:46.789118 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ptml7" Jul 06 19:26:46.789563 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:46.789238 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ptml7" podUID="c2f29af8-1539-48c2-a77e-3b7691f5cea8" Jul 06 19:26:47.788946 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:47.788886 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5xtb" Jul 06 19:26:47.789119 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:47.789005 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5xtb" podUID="867a102b-d74b-4541-8505-4f22e0871dd1" Jul 06 19:26:48.788756 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:48.788729 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ptml7" Jul 06 19:26:48.789276 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:48.788856 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ptml7" podUID="c2f29af8-1539-48c2-a77e-3b7691f5cea8" Jul 06 19:26:48.789276 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:48.788921 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7z7k" Jul 06 19:26:48.789276 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:48.789074 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7z7k" podUID="696eee70-dc27-4b63-b656-ab71bd2ebec5" Jul 06 19:26:49.316945 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:49.316907 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2f29af8-1539-48c2-a77e-3b7691f5cea8-metrics-certs\") pod \"network-metrics-daemon-ptml7\" (UID: \"c2f29af8-1539-48c2-a77e-3b7691f5cea8\") " pod="openshift-multus/network-metrics-daemon-ptml7" Jul 06 19:26:49.317107 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:49.317061 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:26:49.317153 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:49.317132 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2f29af8-1539-48c2-a77e-3b7691f5cea8-metrics-certs podName:c2f29af8-1539-48c2-a77e-3b7691f5cea8 nodeName:}" failed. No retries permitted until 2026-07-06 19:27:05.31711547 +0000 UTC m=+34.137204165 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c2f29af8-1539-48c2-a77e-3b7691f5cea8-metrics-certs") pod "network-metrics-daemon-ptml7" (UID: "c2f29af8-1539-48c2-a77e-3b7691f5cea8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:26:49.417579 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:49.417542 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/696eee70-dc27-4b63-b656-ab71bd2ebec5-original-pull-secret\") pod \"global-pull-secret-syncer-h7z7k\" (UID: \"696eee70-dc27-4b63-b656-ab71bd2ebec5\") " pod="kube-system/global-pull-secret-syncer-h7z7k" Jul 06 19:26:49.417754 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:49.417602 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk9j2\" (UniqueName: \"kubernetes.io/projected/867a102b-d74b-4541-8505-4f22e0871dd1-kube-api-access-hk9j2\") pod \"network-check-target-x5xtb\" (UID: \"867a102b-d74b-4541-8505-4f22e0871dd1\") " pod="openshift-network-diagnostics/network-check-target-x5xtb" Jul 06 19:26:49.417754 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:49.417701 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:26:49.417754 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:49.417722 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:26:49.417754 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:49.417740 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:26:49.417754 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:49.417755 2575 projected.go:196] Error preparing data for projected volume kube-api-access-hk9j2 for pod openshift-network-diagnostics/network-check-target-x5xtb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:26:49.417949 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:49.417769 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/696eee70-dc27-4b63-b656-ab71bd2ebec5-original-pull-secret podName:696eee70-dc27-4b63-b656-ab71bd2ebec5 nodeName:}" failed. No retries permitted until 2026-07-06 19:27:05.417748413 +0000 UTC m=+34.237837105 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/696eee70-dc27-4b63-b656-ab71bd2ebec5-original-pull-secret") pod "global-pull-secret-syncer-h7z7k" (UID: "696eee70-dc27-4b63-b656-ab71bd2ebec5") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:26:49.417949 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:49.417805 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/867a102b-d74b-4541-8505-4f22e0871dd1-kube-api-access-hk9j2 podName:867a102b-d74b-4541-8505-4f22e0871dd1 nodeName:}" failed. No retries permitted until 2026-07-06 19:27:05.417790495 +0000 UTC m=+34.237879226 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hk9j2" (UniqueName: "kubernetes.io/projected/867a102b-d74b-4541-8505-4f22e0871dd1-kube-api-access-hk9j2") pod "network-check-target-x5xtb" (UID: "867a102b-d74b-4541-8505-4f22e0871dd1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:26:49.788116 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:49.788045 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5xtb" Jul 06 19:26:49.788289 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:49.788154 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5xtb" podUID="867a102b-d74b-4541-8505-4f22e0871dd1" Jul 06 19:26:50.788391 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:50.788357 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ptml7" Jul 06 19:26:50.788828 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:50.788405 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7z7k" Jul 06 19:26:50.788828 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:50.788494 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ptml7" podUID="c2f29af8-1539-48c2-a77e-3b7691f5cea8" Jul 06 19:26:50.788828 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:50.788622 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7z7k" podUID="696eee70-dc27-4b63-b656-ab71bd2ebec5" Jul 06 19:26:51.789064 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:51.789033 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5xtb" Jul 06 19:26:51.789523 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:51.789137 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5xtb" podUID="867a102b-d74b-4541-8505-4f22e0871dd1" Jul 06 19:26:52.788330 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:52.788299 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ptml7" Jul 06 19:26:52.788493 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:52.788416 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ptml7" podUID="c2f29af8-1539-48c2-a77e-3b7691f5cea8" Jul 06 19:26:52.788493 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:52.788467 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7z7k" Jul 06 19:26:52.788616 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:52.788570 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7z7k" podUID="696eee70-dc27-4b63-b656-ab71bd2ebec5" Jul 06 19:26:53.789620 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:53.789350 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5xtb" Jul 06 19:26:53.790223 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:53.789695 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5xtb" podUID="867a102b-d74b-4541-8505-4f22e0871dd1" Jul 06 19:26:53.918080 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:53.918052 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q2tnp" event={"ID":"863c8526-b822-48d4-8881-819331146578","Type":"ContainerStarted","Data":"972588852a1bd7905550f4072a2e52b4b76879643a09db31237d0194f501cb13"} Jul 06 19:26:53.919647 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:53.919619 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pgh76" event={"ID":"556124af-4be3-4157-8b4b-70c1c1dfb6fb","Type":"ContainerStarted","Data":"66095506c940ef76586627148341d1cff9af8e70e2ff00a5c56e72fccd19f04f"} Jul 06 19:26:53.921259 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:53.921235 2575 generic.go:356] "Generic (PLEG): container finished" podID="189cc7dd-7ba3-4cfc-b51c-4a8ab328f621" containerID="6e50b7151272b11c91d4cbd1509b017bdc03adc2bdc899a40bee80f2a7a5c662" exitCode=0 Jul 06 19:26:53.921355 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:53.921333 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttjjl" event={"ID":"189cc7dd-7ba3-4cfc-b51c-4a8ab328f621","Type":"ContainerDied","Data":"6e50b7151272b11c91d4cbd1509b017bdc03adc2bdc899a40bee80f2a7a5c662"} Jul 06 19:26:53.923472 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:53.923441 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2xwfj" event={"ID":"9bd815c1-1206-431e-9d76-ba36f79833ac","Type":"ContainerStarted","Data":"262e0916f81222c736281bde3c383989be26063b159a47d07d8c788fa9a945ca"} Jul 06 19:26:53.925341 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:53.925277 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dw4z9" event={"ID":"4e6171d1-c8d8-4800-8998-44acce38b440","Type":"ContainerStarted","Data":"a54309940d4bd68e61b2de2d8f1bc07ba6c6b97dd1f694bb1a4b6056e2ad225e"} Jul 06 19:26:53.930310 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:53.930277 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" event={"ID":"dc44b87e-83bc-4d47-b262-120b31e7e65c","Type":"ContainerStarted","Data":"bc973c61b00f7bc0281d2ab49747deabf7bb9326efcb4f692e523def6918438d"} Jul 06 19:26:53.932089 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:53.932067 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qndvc" event={"ID":"1e221582-b895-4724-8053-b72741d40132","Type":"ContainerStarted","Data":"57a8aef05a0af71b8c62264c28f2b5a8eae8ff0941c737cab3a9d5cc60e4f801"} Jul 06 19:26:53.935126 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:53.935108 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-965cw" event={"ID":"facd7b54-280f-4ee9-91df-3a0773f5dc0a","Type":"ContainerStarted","Data":"83107fa35ed0711d0b3ca16458a5e07d8581135dfad75a6ec061b4d2ec228e0a"} Jul 06 19:26:53.935256 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:53.935190 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-965cw" event={"ID":"facd7b54-280f-4ee9-91df-3a0773f5dc0a","Type":"ContainerStarted","Data":"1224a0269c145dd52fa3ef9864fcbad86ff6a699cda232c6996006ffcf43d162"} Jul 06 19:26:53.935256 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:53.935241 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-965cw" event={"ID":"facd7b54-280f-4ee9-91df-3a0773f5dc0a","Type":"ContainerStarted","Data":"bb1a725a10d40bdaeb2d71141ab50922da81c0696532c25478d1bf8464ef64c7"} Jul 06 19:26:53.935377 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:53.935288 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-965cw" event={"ID":"facd7b54-280f-4ee9-91df-3a0773f5dc0a","Type":"ContainerStarted","Data":"153753cc894f3902cfe1e8f7400d02c18ff9584e7c0883c81760cb0826dedc1a"} Jul 06 19:26:53.935377 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:53.935336 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-965cw" event={"ID":"facd7b54-280f-4ee9-91df-3a0773f5dc0a","Type":"ContainerStarted","Data":"1d76ed5aebdda309cdd9efd4c8d1ab904e3d7d7af9f63ff18ef8ae2acb5da1cf"} Jul 06 19:26:53.936152 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:53.936110 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-q2tnp" podStartSLOduration=4.422744148 podStartE2EDuration="22.936099542s" totalImagesPullingTime="18.513355394s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:26:53.935854764 +0000 UTC m=+22.755943478" watchObservedRunningTime="2026-07-06 19:26:53.936099542 +0000 UTC m=+22.756188253" Jul 06 19:26:53.936408 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:53.936378 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-23.ec2.internal" podStartSLOduration=21.936369548000002 podStartE2EDuration="21.936369548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:26:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:26:36.898906212 +0000 UTC m=+5.718994925" watchObservedRunningTime="2026-07-06 19:26:53.936369548 +0000 UTC m=+22.756458253" Jul 06 19:26:53.954431 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:53.954396 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pgh76" podStartSLOduration=11.437326223 podStartE2EDuration="21.954383586s" totalImagesPullingTime="10.517057363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:26:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:26:53.95352049 +0000 UTC m=+22.773609203" watchObservedRunningTime="2026-07-06 19:26:53.954383586 +0000 UTC m=+22.774472299" Jul 06 19:26:53.976408 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:53.976366 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xlpxn" podStartSLOduration=4.436318747 podStartE2EDuration="22.976349413s" totalImagesPullingTime="18.540030666s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:26:53.975104454 +0000 UTC m=+22.795193163" watchObservedRunningTime="2026-07-06 19:26:53.976349413 +0000 UTC m=+22.796438127" Jul 06 19:26:54.028686 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:54.028609 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-dw4z9" podStartSLOduration=4.435536157 podStartE2EDuration="23.028596286s" totalImagesPullingTime="18.593060129s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:26:54.028101325 +0000 UTC m=+22.848190049" watchObservedRunningTime="2026-07-06 19:26:54.028596286 +0000 UTC m=+22.848684997" Jul 06 19:26:54.048864 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:54.048824 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-2xwfj" podStartSLOduration=3.529975328 podStartE2EDuration="22.048811964s" totalImagesPullingTime="18.518836636s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:26:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:26:54.048396928 +0000 UTC m=+22.868485642" watchObservedRunningTime="2026-07-06 19:26:54.048811964 +0000 UTC m=+22.868900676" Jul 06 19:26:54.756126 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:54.756104 2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 19:26:54.788600 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:54.788575 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7z7k" Jul 06 19:26:54.788714 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:54.788575 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ptml7" Jul 06 19:26:54.788714 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:54.788686 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7z7k" podUID="696eee70-dc27-4b63-b656-ab71bd2ebec5" Jul 06 19:26:54.788830 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:54.788813 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ptml7" podUID="c2f29af8-1539-48c2-a77e-3b7691f5cea8" Jul 06 19:26:54.939362 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:54.939305 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6px9t" event={"ID":"8561ed5c-290c-44b6-8490-8256078f657b","Type":"ContainerStarted","Data":"6e7cd380368ab5839ea536544cd48733ebf4ec59a9258e32f2571a1d27fdc3aa"} Jul 06 19:26:54.941256 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:54.941231 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qndvc" event={"ID":"1e221582-b895-4724-8053-b72741d40132","Type":"ContainerStarted","Data":"7db68b3fb8f3b34ce125763255941eb39f45377ab15a90c7d1cef4aeaf48697c"} Jul 06 19:26:54.944814 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:54.944788 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-965cw" event={"ID":"facd7b54-280f-4ee9-91df-3a0773f5dc0a","Type":"ContainerStarted","Data":"ad09741b7335305dea7cdf2cf0a21edb29bba710e2260a6b9e1359ed1caa5384"} Jul 06 19:26:54.956169 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:54.956131 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6px9t" podStartSLOduration=5.433130496 podStartE2EDuration="23.956118306s" totalImagesPullingTime="18.52298781s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:26:54.955733768 +0000 UTC m=+23.775822479" watchObservedRunningTime="2026-07-06 19:26:54.956118306 +0000 UTC m=+23.776207019" Jul 06 19:26:55.663088 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:55.663051 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pgh76" Jul 06 19:26:55.663616 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:55.663598 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pgh76" Jul 06 19:26:55.667492 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:55.667419 2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T19:26:54.756120492Z","UUID":"bce09af1-0f16-4d54-b28d-44754e5d219e","Handler":null,"Name":"","Endpoint":""} Jul 06 19:26:55.668937 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:55.668918 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 19:26:55.669042 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:55.668952 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 19:26:55.788176 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:55.788146 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5xtb" Jul 06 19:26:55.788316 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:55.788243 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5xtb" podUID="867a102b-d74b-4541-8505-4f22e0871dd1" Jul 06 19:26:55.949304 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:55.949271 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qndvc" event={"ID":"1e221582-b895-4724-8053-b72741d40132","Type":"ContainerStarted","Data":"a14309892e2273e6078ea95c11e6f57a171d5416100783fa5309194857593a3d"} Jul 06 19:26:55.971667 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:55.971622 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qndvc" podStartSLOduration=3.226959286 podStartE2EDuration="23.971607991s" totalImagesPullingTime="20.744648705s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:26:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:26:55.971529687 +0000 UTC m=+24.791618399" watchObservedRunningTime="2026-07-06 19:26:55.971607991 +0000 UTC m=+24.791696700" Jul 06 19:26:56.788417 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:56.788264 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ptml7" Jul 06 19:26:56.788568 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:56.788303 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7z7k" Jul 06 19:26:56.788568 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:56.788488 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ptml7" podUID="c2f29af8-1539-48c2-a77e-3b7691f5cea8" Jul 06 19:26:56.788568 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:56.788554 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7z7k" podUID="696eee70-dc27-4b63-b656-ab71bd2ebec5" Jul 06 19:26:56.955462 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:56.955416 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-965cw" event={"ID":"facd7b54-280f-4ee9-91df-3a0773f5dc0a","Type":"ContainerStarted","Data":"4479daec14bbbd978f57f79e9f2dd683ad755f4ff946bfc8ce9f38ac7cbc8e50"} Jul 06 19:26:56.955462 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:56.955465 2575 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 06 19:26:57.788191 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:57.788147 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5xtb" Jul 06 19:26:57.788371 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:57.788256 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5xtb" podUID="867a102b-d74b-4541-8505-4f22e0871dd1" Jul 06 19:26:58.788021 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:58.787995 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ptml7" Jul 06 19:26:58.788784 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:58.788099 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ptml7" podUID="c2f29af8-1539-48c2-a77e-3b7691f5cea8" Jul 06 19:26:58.788784 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:58.788129 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7z7k" Jul 06 19:26:58.788784 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:58.788198 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7z7k" podUID="696eee70-dc27-4b63-b656-ab71bd2ebec5" Jul 06 19:26:58.962559 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:58.962528 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-965cw" event={"ID":"facd7b54-280f-4ee9-91df-3a0773f5dc0a","Type":"ContainerStarted","Data":"4f237ff8347e4e9db85682b4c8bd00e37ade8a09fdbc0ac966768b9c9a096dca"} Jul 06 19:26:58.962836 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:58.962812 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:58.962965 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:58.962886 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:58.979301 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:58.979099 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:58.999906 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:58.999867 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-965cw" podStartSLOduration=9.377989305 podStartE2EDuration="27.999855285s" totalImagesPullingTime="18.62186598s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:26:58.998565527 +0000 UTC m=+27.818654241" watchObservedRunningTime="2026-07-06 19:26:58.999855285 +0000 UTC m=+27.819943996" Jul 06 19:26:59.789022 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:59.788991 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5xtb" Jul 06 19:26:59.789453 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:26:59.789109 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5xtb" podUID="867a102b-d74b-4541-8505-4f22e0871dd1" Jul 06 19:26:59.966307 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:59.966219 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:26:59.982028 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:26:59.982005 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:27:00.694769 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:00.694731 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h7z7k"] Jul 06 19:27:00.694940 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:00.694853 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7z7k" Jul 06 19:27:00.694989 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:27:00.694958 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7z7k" podUID="696eee70-dc27-4b63-b656-ab71bd2ebec5" Jul 06 19:27:00.695513 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:00.695491 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-x5xtb"] Jul 06 19:27:00.695599 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:00.695570 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5xtb" Jul 06 19:27:00.695696 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:27:00.695671 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5xtb" podUID="867a102b-d74b-4541-8505-4f22e0871dd1" Jul 06 19:27:00.697035 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:00.697012 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ptml7"] Jul 06 19:27:00.697137 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:00.697123 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ptml7" Jul 06 19:27:00.697274 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:27:00.697233 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ptml7" podUID="c2f29af8-1539-48c2-a77e-3b7691f5cea8" Jul 06 19:27:01.188970 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:01.188937 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pgh76" Jul 06 19:27:01.189421 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:01.189063 2575 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 06 19:27:01.189653 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:01.189638 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pgh76" Jul 06 19:27:01.972542 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:01.972317 2575 generic.go:356] "Generic (PLEG): container finished" podID="189cc7dd-7ba3-4cfc-b51c-4a8ab328f621" containerID="e4df0edbc58475fd3cbf5198169efb17739e209a55ec8ae2ee57f0a3cbfa46ec" exitCode=0 Jul 06 19:27:01.972542 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:01.972391 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttjjl" event={"ID":"189cc7dd-7ba3-4cfc-b51c-4a8ab328f621","Type":"ContainerDied","Data":"e4df0edbc58475fd3cbf5198169efb17739e209a55ec8ae2ee57f0a3cbfa46ec"} Jul 06 19:27:02.788668 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:02.788643 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5xtb" Jul 06 19:27:02.789081 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:02.788752 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7z7k" Jul 06 19:27:02.789081 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:02.788764 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ptml7" Jul 06 19:27:02.789081 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:27:02.788754 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5xtb" podUID="867a102b-d74b-4541-8505-4f22e0871dd1" Jul 06 19:27:02.789081 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:27:02.788862 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7z7k" podUID="696eee70-dc27-4b63-b656-ab71bd2ebec5" Jul 06 19:27:02.789081 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:27:02.788957 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ptml7" podUID="c2f29af8-1539-48c2-a77e-3b7691f5cea8" Jul 06 19:27:02.976669 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:02.976642 2575 generic.go:356] "Generic (PLEG): container finished" podID="189cc7dd-7ba3-4cfc-b51c-4a8ab328f621" containerID="a7ad0613413dcb0bebf73d34989b80b4373eb811420abf3ebdfdfcec0a08ea62" exitCode=0 Jul 06 19:27:02.976753 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:02.976685 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttjjl" event={"ID":"189cc7dd-7ba3-4cfc-b51c-4a8ab328f621","Type":"ContainerDied","Data":"a7ad0613413dcb0bebf73d34989b80b4373eb811420abf3ebdfdfcec0a08ea62"} Jul 06 19:27:03.980854 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:03.980778 2575 generic.go:356] "Generic (PLEG): container finished" podID="189cc7dd-7ba3-4cfc-b51c-4a8ab328f621" containerID="dc359bc03a4134b776b84a0e8eb4d00d9009ad97ce7ad483d9948c7f6ceb490b" exitCode=0 Jul 06 19:27:03.980854 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:03.980818 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttjjl" event={"ID":"189cc7dd-7ba3-4cfc-b51c-4a8ab328f621","Type":"ContainerDied","Data":"dc359bc03a4134b776b84a0e8eb4d00d9009ad97ce7ad483d9948c7f6ceb490b"} Jul 06 19:27:04.788393 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:04.788318 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5xtb" Jul 06 19:27:04.788393 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:04.788350 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7z7k" Jul 06 19:27:04.788590 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:04.788414 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ptml7" Jul 06 19:27:04.788590 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:27:04.788422 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5xtb" podUID="867a102b-d74b-4541-8505-4f22e0871dd1" Jul 06 19:27:04.788590 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:27:04.788496 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7z7k" podUID="696eee70-dc27-4b63-b656-ab71bd2ebec5" Jul 06 19:27:04.788590 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:27:04.788568 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ptml7" podUID="c2f29af8-1539-48c2-a77e-3b7691f5cea8" Jul 06 19:27:05.337504 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:05.337468 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2f29af8-1539-48c2-a77e-3b7691f5cea8-metrics-certs\") pod \"network-metrics-daemon-ptml7\" (UID: \"c2f29af8-1539-48c2-a77e-3b7691f5cea8\") " pod="openshift-multus/network-metrics-daemon-ptml7" Jul 06 19:27:05.338033 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:27:05.337578 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:27:05.338033 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:27:05.337634 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2f29af8-1539-48c2-a77e-3b7691f5cea8-metrics-certs podName:c2f29af8-1539-48c2-a77e-3b7691f5cea8 nodeName:}" failed. No retries permitted until 2026-07-06 19:27:37.337618744 +0000 UTC m=+66.157707434 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c2f29af8-1539-48c2-a77e-3b7691f5cea8-metrics-certs") pod "network-metrics-daemon-ptml7" (UID: "c2f29af8-1539-48c2-a77e-3b7691f5cea8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:27:05.437907 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:05.437883 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/696eee70-dc27-4b63-b656-ab71bd2ebec5-original-pull-secret\") pod \"global-pull-secret-syncer-h7z7k\" (UID: \"696eee70-dc27-4b63-b656-ab71bd2ebec5\") " pod="kube-system/global-pull-secret-syncer-h7z7k" Jul 06 19:27:05.438048 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:05.437933 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk9j2\" (UniqueName: \"kubernetes.io/projected/867a102b-d74b-4541-8505-4f22e0871dd1-kube-api-access-hk9j2\") pod \"network-check-target-x5xtb\" (UID: \"867a102b-d74b-4541-8505-4f22e0871dd1\") " pod="openshift-network-diagnostics/network-check-target-x5xtb" Jul 06 19:27:05.438048 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:27:05.438012 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:27:05.438048 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:27:05.438031 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:27:05.438048 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:27:05.438045 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:27:05.438198 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:27:05.438054 2575 projected.go:196] Error preparing data for projected volume kube-api-access-hk9j2 for pod openshift-network-diagnostics/network-check-target-x5xtb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:27:05.438198 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:27:05.438067 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/696eee70-dc27-4b63-b656-ab71bd2ebec5-original-pull-secret podName:696eee70-dc27-4b63-b656-ab71bd2ebec5 nodeName:}" failed. No retries permitted until 2026-07-06 19:27:37.438051644 +0000 UTC m=+66.258140336 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/696eee70-dc27-4b63-b656-ab71bd2ebec5-original-pull-secret") pod "global-pull-secret-syncer-h7z7k" (UID: "696eee70-dc27-4b63-b656-ab71bd2ebec5") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:27:05.438198 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:27:05.438085 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/867a102b-d74b-4541-8505-4f22e0871dd1-kube-api-access-hk9j2 podName:867a102b-d74b-4541-8505-4f22e0871dd1 nodeName:}" failed. No retries permitted until 2026-07-06 19:27:37.438075599 +0000 UTC m=+66.258164290 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hk9j2" (UniqueName: "kubernetes.io/projected/867a102b-d74b-4541-8505-4f22e0871dd1-kube-api-access-hk9j2") pod "network-check-target-x5xtb" (UID: "867a102b-d74b-4541-8505-4f22e0871dd1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:27:05.924845 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:05.924822 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-23.ec2.internal" event="NodeReady" Jul 06 19:27:05.924965 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:05.924953 2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 19:27:05.975030 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:05.975002 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-272x4"] Jul 06 19:27:05.977167 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:05.977147 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-272x4" Jul 06 19:27:05.980276 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:05.980253 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7f4qs\"" Jul 06 19:27:05.980276 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:05.980268 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 19:27:05.980505 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:05.980253 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 19:27:05.984544 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:05.984400 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-96bb8"] Jul 06 19:27:05.990167 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:05.990145 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-96bb8" Jul 06 19:27:05.991737 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:05.991680 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-272x4"] Jul 06 19:27:05.993100 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:05.993075 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 19:27:05.993205 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:05.993112 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 19:27:05.993275 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:05.993219 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 19:27:05.993546 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:05.993531 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kkk98\"" Jul 06 19:27:05.997381 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:05.997295 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-96bb8"] Jul 06 19:27:06.142444 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:06.142418 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0f9bf38-0857-458d-9962-51a2502b2a20-cert\") pod \"ingress-canary-96bb8\" (UID: \"d0f9bf38-0857-458d-9962-51a2502b2a20\") " pod="openshift-ingress-canary/ingress-canary-96bb8" Jul 06 19:27:06.142592 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:06.142457 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1adc5a22-7a57-4da4-9a66-0c451f2603be-metrics-tls\") pod \"dns-default-272x4\" (UID: \"1adc5a22-7a57-4da4-9a66-0c451f2603be\") " pod="openshift-dns/dns-default-272x4" Jul 06 19:27:06.142592 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:06.142509 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1adc5a22-7a57-4da4-9a66-0c451f2603be-tmp-dir\") pod \"dns-default-272x4\" (UID: \"1adc5a22-7a57-4da4-9a66-0c451f2603be\") " pod="openshift-dns/dns-default-272x4" Jul 06 19:27:06.142592 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:06.142539 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52pjh\" (UniqueName: \"kubernetes.io/projected/d0f9bf38-0857-458d-9962-51a2502b2a20-kube-api-access-52pjh\") pod \"ingress-canary-96bb8\" (UID: \"d0f9bf38-0857-458d-9962-51a2502b2a20\") " pod="openshift-ingress-canary/ingress-canary-96bb8" Jul 06 19:27:06.142742 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:06.142599 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1adc5a22-7a57-4da4-9a66-0c451f2603be-config-volume\") pod \"dns-default-272x4\" (UID: \"1adc5a22-7a57-4da4-9a66-0c451f2603be\") " pod="openshift-dns/dns-default-272x4" Jul 06 19:27:06.142742 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:06.142617 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwcmd\" (UniqueName: \"kubernetes.io/projected/1adc5a22-7a57-4da4-9a66-0c451f2603be-kube-api-access-gwcmd\") pod \"dns-default-272x4\" (UID: \"1adc5a22-7a57-4da4-9a66-0c451f2603be\") " pod="openshift-dns/dns-default-272x4" Jul 06 19:27:06.243405 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:06.243383 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1adc5a22-7a57-4da4-9a66-0c451f2603be-tmp-dir\") pod \"dns-default-272x4\" (UID: \"1adc5a22-7a57-4da4-9a66-0c451f2603be\") " pod="openshift-dns/dns-default-272x4" Jul 06 19:27:06.243521 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:06.243413 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52pjh\" (UniqueName: \"kubernetes.io/projected/d0f9bf38-0857-458d-9962-51a2502b2a20-kube-api-access-52pjh\") pod \"ingress-canary-96bb8\" (UID: \"d0f9bf38-0857-458d-9962-51a2502b2a20\") " pod="openshift-ingress-canary/ingress-canary-96bb8" Jul 06 19:27:06.243521 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:06.243463 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1adc5a22-7a57-4da4-9a66-0c451f2603be-config-volume\") pod \"dns-default-272x4\" (UID: \"1adc5a22-7a57-4da4-9a66-0c451f2603be\") " pod="openshift-dns/dns-default-272x4" Jul 06 19:27:06.243521 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:06.243484 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwcmd\" (UniqueName: \"kubernetes.io/projected/1adc5a22-7a57-4da4-9a66-0c451f2603be-kube-api-access-gwcmd\") pod \"dns-default-272x4\" (UID: \"1adc5a22-7a57-4da4-9a66-0c451f2603be\") " pod="openshift-dns/dns-default-272x4" Jul 06 19:27:06.243521 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:06.243515 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0f9bf38-0857-458d-9962-51a2502b2a20-cert\") pod \"ingress-canary-96bb8\" (UID: \"d0f9bf38-0857-458d-9962-51a2502b2a20\") " pod="openshift-ingress-canary/ingress-canary-96bb8" Jul 06 19:27:06.243765 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:06.243546 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1adc5a22-7a57-4da4-9a66-0c451f2603be-metrics-tls\") pod \"dns-default-272x4\" (UID: \"1adc5a22-7a57-4da4-9a66-0c451f2603be\") " pod="openshift-dns/dns-default-272x4" Jul 06 19:27:06.243765 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:27:06.243757 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:27:06.243914 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:06.243892 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1adc5a22-7a57-4da4-9a66-0c451f2603be-tmp-dir\") pod \"dns-default-272x4\" (UID: \"1adc5a22-7a57-4da4-9a66-0c451f2603be\") " pod="openshift-dns/dns-default-272x4" Jul 06 19:27:06.244012 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:27:06.243918 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1adc5a22-7a57-4da4-9a66-0c451f2603be-metrics-tls podName:1adc5a22-7a57-4da4-9a66-0c451f2603be nodeName:}" failed. No retries permitted until 2026-07-06 19:27:06.743899105 +0000 UTC m=+35.563987795 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1adc5a22-7a57-4da4-9a66-0c451f2603be-metrics-tls") pod "dns-default-272x4" (UID: "1adc5a22-7a57-4da4-9a66-0c451f2603be") : secret "dns-default-metrics-tls" not found Jul 06 19:27:06.244214 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:06.244173 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1adc5a22-7a57-4da4-9a66-0c451f2603be-config-volume\") pod \"dns-default-272x4\" (UID: \"1adc5a22-7a57-4da4-9a66-0c451f2603be\") " pod="openshift-dns/dns-default-272x4" Jul 06 19:27:06.247879 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:06.247859 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0f9bf38-0857-458d-9962-51a2502b2a20-cert\") pod \"ingress-canary-96bb8\" (UID: \"d0f9bf38-0857-458d-9962-51a2502b2a20\") " pod="openshift-ingress-canary/ingress-canary-96bb8" Jul 06 19:27:06.254018 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:06.253998 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwcmd\" (UniqueName: \"kubernetes.io/projected/1adc5a22-7a57-4da4-9a66-0c451f2603be-kube-api-access-gwcmd\") pod \"dns-default-272x4\" (UID: \"1adc5a22-7a57-4da4-9a66-0c451f2603be\") " pod="openshift-dns/dns-default-272x4" Jul 06 19:27:06.254225 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:06.254208 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52pjh\" (UniqueName: \"kubernetes.io/projected/d0f9bf38-0857-458d-9962-51a2502b2a20-kube-api-access-52pjh\") pod \"ingress-canary-96bb8\" (UID: \"d0f9bf38-0857-458d-9962-51a2502b2a20\") " pod="openshift-ingress-canary/ingress-canary-96bb8" Jul 06 19:27:06.302280 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:06.302256 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-96bb8" Jul 06 19:27:06.450657 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:06.450441 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-96bb8"] Jul 06 19:27:06.454372 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:27:06.454343 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0f9bf38_0857_458d_9962_51a2502b2a20.slice/crio-2b1d4c01b516580621418f08f32c645c1e6e873192b32ba37731806ac23479a1 WatchSource:0}: Error finding container 2b1d4c01b516580621418f08f32c645c1e6e873192b32ba37731806ac23479a1: Status 404 returned error can't find the container with id 2b1d4c01b516580621418f08f32c645c1e6e873192b32ba37731806ac23479a1 Jul 06 19:27:06.747939 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:06.747851 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1adc5a22-7a57-4da4-9a66-0c451f2603be-metrics-tls\") pod \"dns-default-272x4\" (UID: \"1adc5a22-7a57-4da4-9a66-0c451f2603be\") " pod="openshift-dns/dns-default-272x4" Jul 06 19:27:06.748099 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:27:06.747978 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:27:06.748099 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:27:06.748039 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1adc5a22-7a57-4da4-9a66-0c451f2603be-metrics-tls podName:1adc5a22-7a57-4da4-9a66-0c451f2603be nodeName:}" failed. No retries permitted until 2026-07-06 19:27:07.748025069 +0000 UTC m=+36.568113759 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1adc5a22-7a57-4da4-9a66-0c451f2603be-metrics-tls") pod "dns-default-272x4" (UID: "1adc5a22-7a57-4da4-9a66-0c451f2603be") : secret "dns-default-metrics-tls" not found Jul 06 19:27:06.788354 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:06.788323 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7z7k" Jul 06 19:27:06.788503 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:06.788332 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ptml7" Jul 06 19:27:06.788503 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:06.788431 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5xtb" Jul 06 19:27:06.792387 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:06.792365 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 19:27:06.792552 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:06.792405 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cdwtt\"" Jul 06 19:27:06.792552 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:06.792416 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 19:27:06.792552 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:06.792452 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 19:27:06.792552 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:06.792402 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 19:27:06.792769 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:06.792755 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-t5qdm\"" Jul 06 19:27:06.990784 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:06.990753 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-96bb8" event={"ID":"d0f9bf38-0857-458d-9962-51a2502b2a20","Type":"ContainerStarted","Data":"2b1d4c01b516580621418f08f32c645c1e6e873192b32ba37731806ac23479a1"} Jul 06 19:27:07.754748 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:07.754716 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1adc5a22-7a57-4da4-9a66-0c451f2603be-metrics-tls\") pod \"dns-default-272x4\" (UID: \"1adc5a22-7a57-4da4-9a66-0c451f2603be\") " pod="openshift-dns/dns-default-272x4" Jul 06 19:27:07.755123 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:27:07.754850 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:27:07.755123 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:27:07.754903 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1adc5a22-7a57-4da4-9a66-0c451f2603be-metrics-tls podName:1adc5a22-7a57-4da4-9a66-0c451f2603be nodeName:}" failed. No retries permitted until 2026-07-06 19:27:09.754889323 +0000 UTC m=+38.574978014 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1adc5a22-7a57-4da4-9a66-0c451f2603be-metrics-tls") pod "dns-default-272x4" (UID: "1adc5a22-7a57-4da4-9a66-0c451f2603be") : secret "dns-default-metrics-tls" not found Jul 06 19:27:09.771970 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:09.771929 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1adc5a22-7a57-4da4-9a66-0c451f2603be-metrics-tls\") pod \"dns-default-272x4\" (UID: \"1adc5a22-7a57-4da4-9a66-0c451f2603be\") " pod="openshift-dns/dns-default-272x4" Jul 06 19:27:09.772515 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:27:09.772102 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:27:09.772515 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:27:09.772191 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1adc5a22-7a57-4da4-9a66-0c451f2603be-metrics-tls podName:1adc5a22-7a57-4da4-9a66-0c451f2603be nodeName:}" failed. No retries permitted until 2026-07-06 19:27:13.772157327 +0000 UTC m=+42.592246036 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1adc5a22-7a57-4da4-9a66-0c451f2603be-metrics-tls") pod "dns-default-272x4" (UID: "1adc5a22-7a57-4da4-9a66-0c451f2603be") : secret "dns-default-metrics-tls" not found Jul 06 19:27:11.002701 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:11.002454 2575 generic.go:356] "Generic (PLEG): container finished" podID="189cc7dd-7ba3-4cfc-b51c-4a8ab328f621" containerID="60ff6ec20a5ca2f2bf26309acc7342389c704ef7be64753069194909d40e868a" exitCode=0 Jul 06 19:27:11.003133 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:11.002538 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttjjl" event={"ID":"189cc7dd-7ba3-4cfc-b51c-4a8ab328f621","Type":"ContainerDied","Data":"60ff6ec20a5ca2f2bf26309acc7342389c704ef7be64753069194909d40e868a"} Jul 06 19:27:11.004299 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:11.004267 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-96bb8" event={"ID":"d0f9bf38-0857-458d-9962-51a2502b2a20","Type":"ContainerStarted","Data":"4569b53d9f75122e2b0ddbba75fcbe9e147b00a31e8e64e59bead0d65a00587e"} Jul 06 19:27:11.047659 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:11.047621 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-96bb8" podStartSLOduration=2.276779302 podStartE2EDuration="6.047603158s" totalImagesPullingTime="3.770823856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:27:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:27:11.046330901 +0000 UTC m=+39.866419612" watchObservedRunningTime="2026-07-06 19:27:11.047603158 +0000 UTC m=+39.867691867" Jul 06 19:27:12.009565 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:12.009529 2575 generic.go:356] "Generic (PLEG): container finished" podID="189cc7dd-7ba3-4cfc-b51c-4a8ab328f621" containerID="f1716d23ae14db7809e96fc39df7a562c977e6636242f51439f2864e7f2fcb52" exitCode=0 Jul 06 19:27:12.009963 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:12.009603 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttjjl" event={"ID":"189cc7dd-7ba3-4cfc-b51c-4a8ab328f621","Type":"ContainerDied","Data":"f1716d23ae14db7809e96fc39df7a562c977e6636242f51439f2864e7f2fcb52"} Jul 06 19:27:13.015158 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:13.015124 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttjjl" event={"ID":"189cc7dd-7ba3-4cfc-b51c-4a8ab328f621","Type":"ContainerStarted","Data":"e3a74c063682ce931865bb0e3c53a9ca33cb0d58c39b9a48b252031ae562af3a"} Jul 06 19:27:13.043278 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:13.043234 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ttjjl" podStartSLOduration=8.403026399 podStartE2EDuration="42.043220372s" totalImagesPullingTime="33.640193973s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:26:31 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:27:13.041903026 +0000 UTC m=+41.861991738" watchObservedRunningTime="2026-07-06 19:27:13.043220372 +0000 UTC m=+41.863309083" Jul 06 19:27:13.802789 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:13.802760 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1adc5a22-7a57-4da4-9a66-0c451f2603be-metrics-tls\") pod \"dns-default-272x4\" (UID: \"1adc5a22-7a57-4da4-9a66-0c451f2603be\") " pod="openshift-dns/dns-default-272x4" Jul 06 19:27:13.802931 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:27:13.802904 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:27:13.802968 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:27:13.802960 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1adc5a22-7a57-4da4-9a66-0c451f2603be-metrics-tls podName:1adc5a22-7a57-4da4-9a66-0c451f2603be nodeName:}" failed. No retries permitted until 2026-07-06 19:27:21.802945913 +0000 UTC m=+50.623034608 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1adc5a22-7a57-4da4-9a66-0c451f2603be-metrics-tls") pod "dns-default-272x4" (UID: "1adc5a22-7a57-4da4-9a66-0c451f2603be") : secret "dns-default-metrics-tls" not found Jul 06 19:27:21.846631 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:21.846600 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1adc5a22-7a57-4da4-9a66-0c451f2603be-metrics-tls\") pod \"dns-default-272x4\" (UID: \"1adc5a22-7a57-4da4-9a66-0c451f2603be\") " pod="openshift-dns/dns-default-272x4" Jul 06 19:27:21.847093 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:27:21.846732 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:27:21.847093 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:27:21.846790 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1adc5a22-7a57-4da4-9a66-0c451f2603be-metrics-tls podName:1adc5a22-7a57-4da4-9a66-0c451f2603be nodeName:}" failed. No retries permitted until 2026-07-06 19:27:37.846772509 +0000 UTC m=+66.666861200 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1adc5a22-7a57-4da4-9a66-0c451f2603be-metrics-tls") pod "dns-default-272x4" (UID: "1adc5a22-7a57-4da4-9a66-0c451f2603be") : secret "dns-default-metrics-tls" not found Jul 06 19:27:31.982721 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:31.982694 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-965cw" Jul 06 19:27:37.341724 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:37.341681 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2f29af8-1539-48c2-a77e-3b7691f5cea8-metrics-certs\") pod \"network-metrics-daemon-ptml7\" (UID: \"c2f29af8-1539-48c2-a77e-3b7691f5cea8\") " pod="openshift-multus/network-metrics-daemon-ptml7" Jul 06 19:27:37.344626 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:37.344608 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 19:27:37.352303 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:27:37.352277 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 06 19:27:37.352399 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:27:37.352350 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2f29af8-1539-48c2-a77e-3b7691f5cea8-metrics-certs podName:c2f29af8-1539-48c2-a77e-3b7691f5cea8 nodeName:}" failed. No retries permitted until 2026-07-06 19:28:41.3523345 +0000 UTC m=+130.172423190 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c2f29af8-1539-48c2-a77e-3b7691f5cea8-metrics-certs") pod "network-metrics-daemon-ptml7" (UID: "c2f29af8-1539-48c2-a77e-3b7691f5cea8") : secret "metrics-daemon-secret" not found Jul 06 19:27:37.442886 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:37.442866 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk9j2\" (UniqueName: \"kubernetes.io/projected/867a102b-d74b-4541-8505-4f22e0871dd1-kube-api-access-hk9j2\") pod \"network-check-target-x5xtb\" (UID: \"867a102b-d74b-4541-8505-4f22e0871dd1\") " pod="openshift-network-diagnostics/network-check-target-x5xtb" Jul 06 19:27:37.442966 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:37.442906 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/696eee70-dc27-4b63-b656-ab71bd2ebec5-original-pull-secret\") pod \"global-pull-secret-syncer-h7z7k\" (UID: \"696eee70-dc27-4b63-b656-ab71bd2ebec5\") " pod="kube-system/global-pull-secret-syncer-h7z7k" Jul 06 19:27:37.446724 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:37.446697 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 19:27:37.446724 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:37.446719 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 19:27:37.457433 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:37.457412 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 19:27:37.457630 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:37.457613 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/696eee70-dc27-4b63-b656-ab71bd2ebec5-original-pull-secret\") pod \"global-pull-secret-syncer-h7z7k\" (UID: \"696eee70-dc27-4b63-b656-ab71bd2ebec5\") " pod="kube-system/global-pull-secret-syncer-h7z7k" Jul 06 19:27:37.466913 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:37.466889 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk9j2\" (UniqueName: \"kubernetes.io/projected/867a102b-d74b-4541-8505-4f22e0871dd1-kube-api-access-hk9j2\") pod \"network-check-target-x5xtb\" (UID: \"867a102b-d74b-4541-8505-4f22e0871dd1\") " pod="openshift-network-diagnostics/network-check-target-x5xtb" Jul 06 19:27:37.701143 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:37.701077 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7z7k" Jul 06 19:27:37.710600 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:37.710579 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cdwtt\"" Jul 06 19:27:37.718246 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:37.718225 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5xtb" Jul 06 19:27:37.832545 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:37.832519 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h7z7k"] Jul 06 19:27:37.838076 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:27:37.838051 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod696eee70_dc27_4b63_b656_ab71bd2ebec5.slice/crio-2288d61195f9df5d7dbebffcb8789acd0085373ccc9c2c4e7fc57c137d28f9d2 WatchSource:0}: Error finding container 2288d61195f9df5d7dbebffcb8789acd0085373ccc9c2c4e7fc57c137d28f9d2: Status 404 returned error can't find the container with id 2288d61195f9df5d7dbebffcb8789acd0085373ccc9c2c4e7fc57c137d28f9d2 Jul 06 19:27:37.846298 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:37.846277 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-x5xtb"] Jul 06 19:27:37.848741 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:27:37.848716 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod867a102b_d74b_4541_8505_4f22e0871dd1.slice/crio-6490e47c1a73dceeb41276ac8b09a9112ac8c0781e184c84b5b5b030f6894805 WatchSource:0}: Error finding container 6490e47c1a73dceeb41276ac8b09a9112ac8c0781e184c84b5b5b030f6894805: Status 404 returned error can't find the container with id 6490e47c1a73dceeb41276ac8b09a9112ac8c0781e184c84b5b5b030f6894805 Jul 06 19:27:37.944774 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:37.944751 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1adc5a22-7a57-4da4-9a66-0c451f2603be-metrics-tls\") pod \"dns-default-272x4\" (UID: \"1adc5a22-7a57-4da4-9a66-0c451f2603be\") " pod="openshift-dns/dns-default-272x4" Jul 06 19:27:37.944867 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:27:37.944859 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:27:37.944918 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:27:37.944908 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1adc5a22-7a57-4da4-9a66-0c451f2603be-metrics-tls podName:1adc5a22-7a57-4da4-9a66-0c451f2603be nodeName:}" failed. No retries permitted until 2026-07-06 19:28:09.944891675 +0000 UTC m=+98.764980367 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1adc5a22-7a57-4da4-9a66-0c451f2603be-metrics-tls") pod "dns-default-272x4" (UID: "1adc5a22-7a57-4da4-9a66-0c451f2603be") : secret "dns-default-metrics-tls" not found Jul 06 19:27:38.072670 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:38.072637 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-x5xtb" event={"ID":"867a102b-d74b-4541-8505-4f22e0871dd1","Type":"ContainerStarted","Data":"6490e47c1a73dceeb41276ac8b09a9112ac8c0781e184c84b5b5b030f6894805"} Jul 06 19:27:38.073605 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:38.073588 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-h7z7k" event={"ID":"696eee70-dc27-4b63-b656-ab71bd2ebec5","Type":"ContainerStarted","Data":"2288d61195f9df5d7dbebffcb8789acd0085373ccc9c2c4e7fc57c137d28f9d2"} Jul 06 19:27:43.088262 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:43.088226 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-x5xtb" event={"ID":"867a102b-d74b-4541-8505-4f22e0871dd1","Type":"ContainerStarted","Data":"17d7bde841a13d1cb27632ba5f6251b981385cd0a54e460970504b6ecb1a9c47"} Jul 06 19:27:43.088770 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:43.088310 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-x5xtb" Jul 06 19:27:43.089737 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:43.089714 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-h7z7k" event={"ID":"696eee70-dc27-4b63-b656-ab71bd2ebec5","Type":"ContainerStarted","Data":"ef0e8c163f1291846724fc88a5a467135d17c35f43bf5d92fbeebaec99ed7ad3"} Jul 06 19:27:43.106694 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:43.106653 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-x5xtb" podStartSLOduration=67.390368984 podStartE2EDuration="1m12.106639306s" totalImagesPullingTime="4.716270322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:27:43.105758314 +0000 UTC m=+71.925847026" watchObservedRunningTime="2026-07-06 19:27:43.106639306 +0000 UTC m=+71.926728018" Jul 06 19:27:43.124560 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:27:43.122022 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-h7z7k" podStartSLOduration=66.390361775 podStartE2EDuration="1m11.122004592s" totalImagesPullingTime="4.731642817s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:26:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:27:43.121400956 +0000 UTC m=+71.941489669" watchObservedRunningTime="2026-07-06 19:27:43.122004592 +0000 UTC m=+71.942093306" Jul 06 19:28:09.951761 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:09.951724 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1adc5a22-7a57-4da4-9a66-0c451f2603be-metrics-tls\") pod \"dns-default-272x4\" (UID: \"1adc5a22-7a57-4da4-9a66-0c451f2603be\") " pod="openshift-dns/dns-default-272x4" Jul 06 19:28:09.952290 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:09.951846 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:28:09.952290 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:09.951907 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1adc5a22-7a57-4da4-9a66-0c451f2603be-metrics-tls podName:1adc5a22-7a57-4da4-9a66-0c451f2603be nodeName:}" failed. No retries permitted until 2026-07-06 19:29:13.951893407 +0000 UTC m=+162.771982096 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1adc5a22-7a57-4da4-9a66-0c451f2603be-metrics-tls") pod "dns-default-272x4" (UID: "1adc5a22-7a57-4da4-9a66-0c451f2603be") : secret "dns-default-metrics-tls" not found Jul 06 19:28:12.806036 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.806004 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-5wmbj"] Jul 06 19:28:12.838430 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.838405 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6dcdb5648f-whl8g"] Jul 06 19:28:12.838557 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.838524 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-5wmbj" Jul 06 19:28:12.841137 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.841117 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 06 19:28:12.841804 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.841785 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 19:28:12.841875 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.841813 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 19:28:12.843059 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.843040 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-rhr8m\"" Jul 06 19:28:12.843159 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.843093 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 06 19:28:12.852294 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.852274 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-5wmbj"] Jul 06 19:28:12.852378 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.852299 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6dcdb5648f-whl8g"] Jul 06 19:28:12.852378 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.852358 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6dcdb5648f-whl8g" Jul 06 19:28:12.854834 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.854818 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 19:28:12.854961 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.854946 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 06 19:28:12.854961 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.854953 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 06 19:28:12.855278 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.855262 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 06 19:28:12.855366 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.855323 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 19:28:12.855366 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.855328 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-jjtj8\"" Jul 06 19:28:12.855509 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.855493 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 06 19:28:12.861143 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.861126 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 06 19:28:12.870487 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.870470 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/677b6ba1-3252-4777-88fb-b4238918b699-stats-auth\") pod \"router-default-6dcdb5648f-whl8g\" (UID: \"677b6ba1-3252-4777-88fb-b4238918b699\") " pod="openshift-ingress/router-default-6dcdb5648f-whl8g" Jul 06 19:28:12.870580 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.870507 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3efc9cc-6d76-4cde-a12a-c0f5022e1abf-serving-cert\") pod \"insights-operator-5c847dcfc4-5wmbj\" (UID: \"e3efc9cc-6d76-4cde-a12a-c0f5022e1abf\") " pod="openshift-insights/insights-operator-5c847dcfc4-5wmbj" Jul 06 19:28:12.870580 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.870528 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmljs\" (UniqueName: \"kubernetes.io/projected/677b6ba1-3252-4777-88fb-b4238918b699-kube-api-access-wmljs\") pod \"router-default-6dcdb5648f-whl8g\" (UID: \"677b6ba1-3252-4777-88fb-b4238918b699\") " pod="openshift-ingress/router-default-6dcdb5648f-whl8g" Jul 06 19:28:12.870580 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.870550 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3efc9cc-6d76-4cde-a12a-c0f5022e1abf-tmp\") pod \"insights-operator-5c847dcfc4-5wmbj\" (UID: \"e3efc9cc-6d76-4cde-a12a-c0f5022e1abf\") " pod="openshift-insights/insights-operator-5c847dcfc4-5wmbj" Jul 06 19:28:12.870693 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.870607 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpks8\" (UniqueName: \"kubernetes.io/projected/e3efc9cc-6d76-4cde-a12a-c0f5022e1abf-kube-api-access-mpks8\") pod \"insights-operator-5c847dcfc4-5wmbj\" (UID: \"e3efc9cc-6d76-4cde-a12a-c0f5022e1abf\") " pod="openshift-insights/insights-operator-5c847dcfc4-5wmbj" Jul 06 19:28:12.870693 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.870628 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/677b6ba1-3252-4777-88fb-b4238918b699-default-certificate\") pod \"router-default-6dcdb5648f-whl8g\" (UID: \"677b6ba1-3252-4777-88fb-b4238918b699\") " pod="openshift-ingress/router-default-6dcdb5648f-whl8g" Jul 06 19:28:12.870693 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.870668 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e3efc9cc-6d76-4cde-a12a-c0f5022e1abf-snapshots\") pod \"insights-operator-5c847dcfc4-5wmbj\" (UID: \"e3efc9cc-6d76-4cde-a12a-c0f5022e1abf\") " pod="openshift-insights/insights-operator-5c847dcfc4-5wmbj" Jul 06 19:28:12.870693 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.870686 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/677b6ba1-3252-4777-88fb-b4238918b699-metrics-certs\") pod \"router-default-6dcdb5648f-whl8g\" (UID: \"677b6ba1-3252-4777-88fb-b4238918b699\") " pod="openshift-ingress/router-default-6dcdb5648f-whl8g" Jul 06 19:28:12.870838 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.870703 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3efc9cc-6d76-4cde-a12a-c0f5022e1abf-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-5wmbj\" (UID: \"e3efc9cc-6d76-4cde-a12a-c0f5022e1abf\") " pod="openshift-insights/insights-operator-5c847dcfc4-5wmbj" Jul 06 19:28:12.870838 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.870717 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3efc9cc-6d76-4cde-a12a-c0f5022e1abf-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-5wmbj\" (UID: \"e3efc9cc-6d76-4cde-a12a-c0f5022e1abf\") " pod="openshift-insights/insights-operator-5c847dcfc4-5wmbj" Jul 06 19:28:12.870838 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.870734 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/677b6ba1-3252-4777-88fb-b4238918b699-service-ca-bundle\") pod \"router-default-6dcdb5648f-whl8g\" (UID: \"677b6ba1-3252-4777-88fb-b4238918b699\") " pod="openshift-ingress/router-default-6dcdb5648f-whl8g" Jul 06 19:28:12.971255 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.971230 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpks8\" (UniqueName: \"kubernetes.io/projected/e3efc9cc-6d76-4cde-a12a-c0f5022e1abf-kube-api-access-mpks8\") pod \"insights-operator-5c847dcfc4-5wmbj\" (UID: \"e3efc9cc-6d76-4cde-a12a-c0f5022e1abf\") " pod="openshift-insights/insights-operator-5c847dcfc4-5wmbj" Jul 06 19:28:12.971338 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.971256 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/677b6ba1-3252-4777-88fb-b4238918b699-default-certificate\") pod \"router-default-6dcdb5648f-whl8g\" (UID: \"677b6ba1-3252-4777-88fb-b4238918b699\") " pod="openshift-ingress/router-default-6dcdb5648f-whl8g" Jul 06 19:28:12.971338 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.971289 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e3efc9cc-6d76-4cde-a12a-c0f5022e1abf-snapshots\") pod \"insights-operator-5c847dcfc4-5wmbj\" (UID: \"e3efc9cc-6d76-4cde-a12a-c0f5022e1abf\") " pod="openshift-insights/insights-operator-5c847dcfc4-5wmbj" Jul 06 19:28:12.971415 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.971391 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/677b6ba1-3252-4777-88fb-b4238918b699-metrics-certs\") pod \"router-default-6dcdb5648f-whl8g\" (UID: \"677b6ba1-3252-4777-88fb-b4238918b699\") " pod="openshift-ingress/router-default-6dcdb5648f-whl8g" Jul 06 19:28:12.971460 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.971422 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3efc9cc-6d76-4cde-a12a-c0f5022e1abf-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-5wmbj\" (UID: \"e3efc9cc-6d76-4cde-a12a-c0f5022e1abf\") " pod="openshift-insights/insights-operator-5c847dcfc4-5wmbj" Jul 06 19:28:12.971460 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.971437 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3efc9cc-6d76-4cde-a12a-c0f5022e1abf-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-5wmbj\" (UID: \"e3efc9cc-6d76-4cde-a12a-c0f5022e1abf\") " pod="openshift-insights/insights-operator-5c847dcfc4-5wmbj" Jul 06 19:28:12.971551 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.971456 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/677b6ba1-3252-4777-88fb-b4238918b699-service-ca-bundle\") pod \"router-default-6dcdb5648f-whl8g\" (UID: \"677b6ba1-3252-4777-88fb-b4238918b699\") " pod="openshift-ingress/router-default-6dcdb5648f-whl8g" Jul 06 19:28:12.971551 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.971486 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/677b6ba1-3252-4777-88fb-b4238918b699-stats-auth\") pod \"router-default-6dcdb5648f-whl8g\" (UID: \"677b6ba1-3252-4777-88fb-b4238918b699\") " pod="openshift-ingress/router-default-6dcdb5648f-whl8g" Jul 06 19:28:12.971551 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.971528 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3efc9cc-6d76-4cde-a12a-c0f5022e1abf-serving-cert\") pod \"insights-operator-5c847dcfc4-5wmbj\" (UID: \"e3efc9cc-6d76-4cde-a12a-c0f5022e1abf\") " pod="openshift-insights/insights-operator-5c847dcfc4-5wmbj" Jul 06 19:28:12.971693 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:12.971547 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 19:28:12.971693 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.971563 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmljs\" (UniqueName: \"kubernetes.io/projected/677b6ba1-3252-4777-88fb-b4238918b699-kube-api-access-wmljs\") pod \"router-default-6dcdb5648f-whl8g\" (UID: \"677b6ba1-3252-4777-88fb-b4238918b699\") " pod="openshift-ingress/router-default-6dcdb5648f-whl8g" Jul 06 19:28:12.971693 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:12.971589 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/677b6ba1-3252-4777-88fb-b4238918b699-service-ca-bundle podName:677b6ba1-3252-4777-88fb-b4238918b699 nodeName:}" failed. No retries permitted until 2026-07-06 19:28:13.471570231 +0000 UTC m=+102.291658940 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/677b6ba1-3252-4777-88fb-b4238918b699-service-ca-bundle") pod "router-default-6dcdb5648f-whl8g" (UID: "677b6ba1-3252-4777-88fb-b4238918b699") : configmap references non-existent config key: service-ca.crt Jul 06 19:28:12.971693 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:12.971612 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/677b6ba1-3252-4777-88fb-b4238918b699-metrics-certs podName:677b6ba1-3252-4777-88fb-b4238918b699 nodeName:}" failed. No retries permitted until 2026-07-06 19:28:13.471602026 +0000 UTC m=+102.291690718 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/677b6ba1-3252-4777-88fb-b4238918b699-metrics-certs") pod "router-default-6dcdb5648f-whl8g" (UID: "677b6ba1-3252-4777-88fb-b4238918b699") : secret "router-metrics-certs-default" not found Jul 06 19:28:12.971693 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.971641 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3efc9cc-6d76-4cde-a12a-c0f5022e1abf-tmp\") pod \"insights-operator-5c847dcfc4-5wmbj\" (UID: \"e3efc9cc-6d76-4cde-a12a-c0f5022e1abf\") " pod="openshift-insights/insights-operator-5c847dcfc4-5wmbj" Jul 06 19:28:12.971925 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.971908 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3efc9cc-6d76-4cde-a12a-c0f5022e1abf-tmp\") pod \"insights-operator-5c847dcfc4-5wmbj\" (UID: \"e3efc9cc-6d76-4cde-a12a-c0f5022e1abf\") " pod="openshift-insights/insights-operator-5c847dcfc4-5wmbj" Jul 06 19:28:12.971993 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.971970 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e3efc9cc-6d76-4cde-a12a-c0f5022e1abf-snapshots\") pod \"insights-operator-5c847dcfc4-5wmbj\" (UID: \"e3efc9cc-6d76-4cde-a12a-c0f5022e1abf\") " pod="openshift-insights/insights-operator-5c847dcfc4-5wmbj" Jul 06 19:28:12.972103 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.972087 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3efc9cc-6d76-4cde-a12a-c0f5022e1abf-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-5wmbj\" (UID: \"e3efc9cc-6d76-4cde-a12a-c0f5022e1abf\") " pod="openshift-insights/insights-operator-5c847dcfc4-5wmbj" Jul 06 19:28:12.972436 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.972416 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3efc9cc-6d76-4cde-a12a-c0f5022e1abf-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-5wmbj\" (UID: \"e3efc9cc-6d76-4cde-a12a-c0f5022e1abf\") " pod="openshift-insights/insights-operator-5c847dcfc4-5wmbj" Jul 06 19:28:12.973682 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.973665 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/677b6ba1-3252-4777-88fb-b4238918b699-default-certificate\") pod \"router-default-6dcdb5648f-whl8g\" (UID: \"677b6ba1-3252-4777-88fb-b4238918b699\") " pod="openshift-ingress/router-default-6dcdb5648f-whl8g" Jul 06 19:28:12.973752 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.973689 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3efc9cc-6d76-4cde-a12a-c0f5022e1abf-serving-cert\") pod \"insights-operator-5c847dcfc4-5wmbj\" (UID: \"e3efc9cc-6d76-4cde-a12a-c0f5022e1abf\") " pod="openshift-insights/insights-operator-5c847dcfc4-5wmbj" Jul 06 19:28:12.973985 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.973969 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/677b6ba1-3252-4777-88fb-b4238918b699-stats-auth\") pod \"router-default-6dcdb5648f-whl8g\" (UID: \"677b6ba1-3252-4777-88fb-b4238918b699\") " pod="openshift-ingress/router-default-6dcdb5648f-whl8g" Jul 06 19:28:12.983139 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.982339 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpks8\" (UniqueName: \"kubernetes.io/projected/e3efc9cc-6d76-4cde-a12a-c0f5022e1abf-kube-api-access-mpks8\") pod \"insights-operator-5c847dcfc4-5wmbj\" (UID: \"e3efc9cc-6d76-4cde-a12a-c0f5022e1abf\") " pod="openshift-insights/insights-operator-5c847dcfc4-5wmbj" Jul 06 19:28:12.985715 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:12.983867 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmljs\" (UniqueName: \"kubernetes.io/projected/677b6ba1-3252-4777-88fb-b4238918b699-kube-api-access-wmljs\") pod \"router-default-6dcdb5648f-whl8g\" (UID: \"677b6ba1-3252-4777-88fb-b4238918b699\") " pod="openshift-ingress/router-default-6dcdb5648f-whl8g" Jul 06 19:28:13.146816 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:13.146799 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-5wmbj" Jul 06 19:28:13.298514 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:13.298490 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-5wmbj"] Jul 06 19:28:13.301083 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:28:13.301055 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3efc9cc_6d76_4cde_a12a_c0f5022e1abf.slice/crio-3ca39466fce316d9101d853a8423bc7e41c18397d750498cd968eeca8d9acfd7 WatchSource:0}: Error finding container 3ca39466fce316d9101d853a8423bc7e41c18397d750498cd968eeca8d9acfd7: Status 404 returned error can't find the container with id 3ca39466fce316d9101d853a8423bc7e41c18397d750498cd968eeca8d9acfd7 Jul 06 19:28:13.475596 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:13.475540 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/677b6ba1-3252-4777-88fb-b4238918b699-service-ca-bundle\") pod \"router-default-6dcdb5648f-whl8g\" (UID: \"677b6ba1-3252-4777-88fb-b4238918b699\") " pod="openshift-ingress/router-default-6dcdb5648f-whl8g" Jul 06 19:28:13.475685 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:13.475607 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/677b6ba1-3252-4777-88fb-b4238918b699-metrics-certs\") pod \"router-default-6dcdb5648f-whl8g\" (UID: \"677b6ba1-3252-4777-88fb-b4238918b699\") " pod="openshift-ingress/router-default-6dcdb5648f-whl8g" Jul 06 19:28:13.475721 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:13.475690 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/677b6ba1-3252-4777-88fb-b4238918b699-service-ca-bundle podName:677b6ba1-3252-4777-88fb-b4238918b699 nodeName:}" failed. No retries permitted until 2026-07-06 19:28:14.475672861 +0000 UTC m=+103.295761550 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/677b6ba1-3252-4777-88fb-b4238918b699-service-ca-bundle") pod "router-default-6dcdb5648f-whl8g" (UID: "677b6ba1-3252-4777-88fb-b4238918b699") : configmap references non-existent config key: service-ca.crt Jul 06 19:28:13.475721 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:13.475699 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 19:28:13.475791 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:13.475737 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/677b6ba1-3252-4777-88fb-b4238918b699-metrics-certs podName:677b6ba1-3252-4777-88fb-b4238918b699 nodeName:}" failed. No retries permitted until 2026-07-06 19:28:14.475725283 +0000 UTC m=+103.295813973 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/677b6ba1-3252-4777-88fb-b4238918b699-metrics-certs") pod "router-default-6dcdb5648f-whl8g" (UID: "677b6ba1-3252-4777-88fb-b4238918b699") : secret "router-metrics-certs-default" not found Jul 06 19:28:14.094536 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:14.094511 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-x5xtb" Jul 06 19:28:14.159631 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:14.159594 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-5wmbj" event={"ID":"e3efc9cc-6d76-4cde-a12a-c0f5022e1abf","Type":"ContainerStarted","Data":"3ca39466fce316d9101d853a8423bc7e41c18397d750498cd968eeca8d9acfd7"} Jul 06 19:28:14.483742 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:14.483666 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/677b6ba1-3252-4777-88fb-b4238918b699-metrics-certs\") pod \"router-default-6dcdb5648f-whl8g\" (UID: \"677b6ba1-3252-4777-88fb-b4238918b699\") " pod="openshift-ingress/router-default-6dcdb5648f-whl8g" Jul 06 19:28:14.483742 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:14.483711 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/677b6ba1-3252-4777-88fb-b4238918b699-service-ca-bundle\") pod \"router-default-6dcdb5648f-whl8g\" (UID: \"677b6ba1-3252-4777-88fb-b4238918b699\") " pod="openshift-ingress/router-default-6dcdb5648f-whl8g" Jul 06 19:28:14.483962 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:14.483822 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 19:28:14.483962 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:14.483868 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/677b6ba1-3252-4777-88fb-b4238918b699-service-ca-bundle podName:677b6ba1-3252-4777-88fb-b4238918b699 nodeName:}" failed. No retries permitted until 2026-07-06 19:28:16.483850648 +0000 UTC m=+105.303939344 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/677b6ba1-3252-4777-88fb-b4238918b699-service-ca-bundle") pod "router-default-6dcdb5648f-whl8g" (UID: "677b6ba1-3252-4777-88fb-b4238918b699") : configmap references non-existent config key: service-ca.crt Jul 06 19:28:14.483962 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:14.483889 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/677b6ba1-3252-4777-88fb-b4238918b699-metrics-certs podName:677b6ba1-3252-4777-88fb-b4238918b699 nodeName:}" failed. No retries permitted until 2026-07-06 19:28:16.483881413 +0000 UTC m=+105.303970110 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/677b6ba1-3252-4777-88fb-b4238918b699-metrics-certs") pod "router-default-6dcdb5648f-whl8g" (UID: "677b6ba1-3252-4777-88fb-b4238918b699") : secret "router-metrics-certs-default" not found Jul 06 19:28:14.659692 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:14.659661 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-b8955"] Jul 06 19:28:14.680358 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:14.680312 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-b8955"] Jul 06 19:28:14.680511 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:14.680411 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-b8955" Jul 06 19:28:14.683538 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:14.683494 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 06 19:28:14.683538 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:14.683512 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-vlx4w\"" Jul 06 19:28:14.684560 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:14.684540 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:28:14.761033 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:14.760983 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-kxqkc"] Jul 06 19:28:14.780749 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:14.780720 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-kxqkc"] Jul 06 19:28:14.780862 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:14.780845 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-kxqkc" Jul 06 19:28:14.783687 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:14.783665 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 06 19:28:14.783791 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:14.783752 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:28:14.783852 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:14.783820 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 06 19:28:14.783903 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:14.783881 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-h45tb\"" Jul 06 19:28:14.785211 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:14.785175 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwp8g\" (UniqueName: \"kubernetes.io/projected/70caece2-9e4c-4518-9a0d-b36a8cc17d50-kube-api-access-zwp8g\") pod \"volume-data-source-validator-5d456fdbcd-b8955\" (UID: \"70caece2-9e4c-4518-9a0d-b36a8cc17d50\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-b8955" Jul 06 19:28:14.886381 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:14.886353 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwp8g\" (UniqueName: \"kubernetes.io/projected/70caece2-9e4c-4518-9a0d-b36a8cc17d50-kube-api-access-zwp8g\") pod \"volume-data-source-validator-5d456fdbcd-b8955\" (UID: \"70caece2-9e4c-4518-9a0d-b36a8cc17d50\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-b8955" Jul 06 19:28:14.886537 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:14.886399 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlbb4\" (UniqueName: \"kubernetes.io/projected/55480535-e86a-460e-b1bf-5c8da9c2ffa3-kube-api-access-dlbb4\") pod \"cluster-samples-operator-56fcf86bf-kxqkc\" (UID: \"55480535-e86a-460e-b1bf-5c8da9c2ffa3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-kxqkc" Jul 06 19:28:14.886537 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:14.886431 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/55480535-e86a-460e-b1bf-5c8da9c2ffa3-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-kxqkc\" (UID: \"55480535-e86a-460e-b1bf-5c8da9c2ffa3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-kxqkc" Jul 06 19:28:14.897258 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:14.897237 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwp8g\" (UniqueName: \"kubernetes.io/projected/70caece2-9e4c-4518-9a0d-b36a8cc17d50-kube-api-access-zwp8g\") pod \"volume-data-source-validator-5d456fdbcd-b8955\" (UID: \"70caece2-9e4c-4518-9a0d-b36a8cc17d50\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-b8955" Jul 06 19:28:14.987422 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:14.987391 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlbb4\" (UniqueName: \"kubernetes.io/projected/55480535-e86a-460e-b1bf-5c8da9c2ffa3-kube-api-access-dlbb4\") pod \"cluster-samples-operator-56fcf86bf-kxqkc\" (UID: \"55480535-e86a-460e-b1bf-5c8da9c2ffa3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-kxqkc" Jul 06 19:28:14.987565 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:14.987434 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/55480535-e86a-460e-b1bf-5c8da9c2ffa3-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-kxqkc\" (UID: \"55480535-e86a-460e-b1bf-5c8da9c2ffa3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-kxqkc" Jul 06 19:28:14.987565 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:14.987556 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 19:28:14.987677 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:14.987627 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55480535-e86a-460e-b1bf-5c8da9c2ffa3-samples-operator-tls podName:55480535-e86a-460e-b1bf-5c8da9c2ffa3 nodeName:}" failed. No retries permitted until 2026-07-06 19:28:15.48760757 +0000 UTC m=+104.307696262 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/55480535-e86a-460e-b1bf-5c8da9c2ffa3-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-kxqkc" (UID: "55480535-e86a-460e-b1bf-5c8da9c2ffa3") : secret "samples-operator-tls" not found Jul 06 19:28:14.991332 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:14.991309 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-b8955" Jul 06 19:28:15.002105 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:15.002077 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlbb4\" (UniqueName: \"kubernetes.io/projected/55480535-e86a-460e-b1bf-5c8da9c2ffa3-kube-api-access-dlbb4\") pod \"cluster-samples-operator-56fcf86bf-kxqkc\" (UID: \"55480535-e86a-460e-b1bf-5c8da9c2ffa3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-kxqkc" Jul 06 19:28:15.491739 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:15.491715 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/55480535-e86a-460e-b1bf-5c8da9c2ffa3-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-kxqkc\" (UID: \"55480535-e86a-460e-b1bf-5c8da9c2ffa3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-kxqkc" Jul 06 19:28:15.492071 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:15.491838 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 19:28:15.492071 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:15.491898 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55480535-e86a-460e-b1bf-5c8da9c2ffa3-samples-operator-tls podName:55480535-e86a-460e-b1bf-5c8da9c2ffa3 nodeName:}" failed. No retries permitted until 2026-07-06 19:28:16.491880394 +0000 UTC m=+105.311969083 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/55480535-e86a-460e-b1bf-5c8da9c2ffa3-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-kxqkc" (UID: "55480535-e86a-460e-b1bf-5c8da9c2ffa3") : secret "samples-operator-tls" not found Jul 06 19:28:15.496094 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:15.496067 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-b8955"] Jul 06 19:28:15.655145 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:28:15.655080 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70caece2_9e4c_4518_9a0d_b36a8cc17d50.slice/crio-8ebe06333829d95d810b26111d8a12aca5f4ce18b74d5c329c71409f86de63d1 WatchSource:0}: Error finding container 8ebe06333829d95d810b26111d8a12aca5f4ce18b74d5c329c71409f86de63d1: Status 404 returned error can't find the container with id 8ebe06333829d95d810b26111d8a12aca5f4ce18b74d5c329c71409f86de63d1 Jul 06 19:28:16.166673 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:16.166641 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-5wmbj" event={"ID":"e3efc9cc-6d76-4cde-a12a-c0f5022e1abf","Type":"ContainerStarted","Data":"fe09003cef97f6a6d47f53f334ca46c09cd53d800d6dc55bbceef00b402aebdd"} Jul 06 19:28:16.168126 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:16.168102 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-b8955" event={"ID":"70caece2-9e4c-4518-9a0d-b36a8cc17d50","Type":"ContainerStarted","Data":"8ebe06333829d95d810b26111d8a12aca5f4ce18b74d5c329c71409f86de63d1"} Jul 06 19:28:16.188484 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:16.188437 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-5wmbj" podStartSLOduration=1.808631477 podStartE2EDuration="4.188421399s" totalImagesPullingTime="2.379789922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:28:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:28:16.187494185 +0000 UTC m=+105.007582896" watchObservedRunningTime="2026-07-06 19:28:16.188421399 +0000 UTC m=+105.008510144" Jul 06 19:28:16.460117 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:16.459983 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gnxgr"] Jul 06 19:28:16.463601 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:16.463553 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gnxgr" Jul 06 19:28:16.469138 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:16.468962 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 19:28:16.469281 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:16.469143 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 19:28:16.469281 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:16.469176 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k82dj\"" Jul 06 19:28:16.484024 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:16.484000 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gnxgr"] Jul 06 19:28:16.498399 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:16.498377 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/677b6ba1-3252-4777-88fb-b4238918b699-metrics-certs\") pod \"router-default-6dcdb5648f-whl8g\" (UID: \"677b6ba1-3252-4777-88fb-b4238918b699\") " pod="openshift-ingress/router-default-6dcdb5648f-whl8g" Jul 06 19:28:16.498747 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:16.498413 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/660f827d-c694-49d5-9af0-1458f46f2ffb-data-volume\") pod \"insights-runtime-extractor-gnxgr\" (UID: \"660f827d-c694-49d5-9af0-1458f46f2ffb\") " pod="openshift-insights/insights-runtime-extractor-gnxgr" Jul 06 19:28:16.498747 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:16.498448 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/677b6ba1-3252-4777-88fb-b4238918b699-service-ca-bundle\") pod \"router-default-6dcdb5648f-whl8g\" (UID: \"677b6ba1-3252-4777-88fb-b4238918b699\") " pod="openshift-ingress/router-default-6dcdb5648f-whl8g" Jul 06 19:28:16.498747 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:16.498475 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzcb4\" (UniqueName: \"kubernetes.io/projected/660f827d-c694-49d5-9af0-1458f46f2ffb-kube-api-access-zzcb4\") pod \"insights-runtime-extractor-gnxgr\" (UID: \"660f827d-c694-49d5-9af0-1458f46f2ffb\") " pod="openshift-insights/insights-runtime-extractor-gnxgr" Jul 06 19:28:16.498747 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:16.498495 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 19:28:16.498747 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:16.498506 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/55480535-e86a-460e-b1bf-5c8da9c2ffa3-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-kxqkc\" (UID: \"55480535-e86a-460e-b1bf-5c8da9c2ffa3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-kxqkc" Jul 06 19:28:16.498747 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:16.498555 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/677b6ba1-3252-4777-88fb-b4238918b699-metrics-certs podName:677b6ba1-3252-4777-88fb-b4238918b699 nodeName:}" failed. No retries permitted until 2026-07-06 19:28:20.498540683 +0000 UTC m=+109.318629377 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/677b6ba1-3252-4777-88fb-b4238918b699-metrics-certs") pod "router-default-6dcdb5648f-whl8g" (UID: "677b6ba1-3252-4777-88fb-b4238918b699") : secret "router-metrics-certs-default" not found Jul 06 19:28:16.498747 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:16.498599 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/660f827d-c694-49d5-9af0-1458f46f2ffb-crio-socket\") pod \"insights-runtime-extractor-gnxgr\" (UID: \"660f827d-c694-49d5-9af0-1458f46f2ffb\") " pod="openshift-insights/insights-runtime-extractor-gnxgr" Jul 06 19:28:16.498747 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:16.498615 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 19:28:16.498747 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:16.498629 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/677b6ba1-3252-4777-88fb-b4238918b699-service-ca-bundle podName:677b6ba1-3252-4777-88fb-b4238918b699 nodeName:}" failed. No retries permitted until 2026-07-06 19:28:20.4986108 +0000 UTC m=+109.318699502 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/677b6ba1-3252-4777-88fb-b4238918b699-service-ca-bundle") pod "router-default-6dcdb5648f-whl8g" (UID: "677b6ba1-3252-4777-88fb-b4238918b699") : configmap references non-existent config key: service-ca.crt Jul 06 19:28:16.498747 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:16.498659 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55480535-e86a-460e-b1bf-5c8da9c2ffa3-samples-operator-tls podName:55480535-e86a-460e-b1bf-5c8da9c2ffa3 nodeName:}" failed. No retries permitted until 2026-07-06 19:28:18.498645825 +0000 UTC m=+107.318734519 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/55480535-e86a-460e-b1bf-5c8da9c2ffa3-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-kxqkc" (UID: "55480535-e86a-460e-b1bf-5c8da9c2ffa3") : secret "samples-operator-tls" not found Jul 06 19:28:16.498747 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:16.498656 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/660f827d-c694-49d5-9af0-1458f46f2ffb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gnxgr\" (UID: \"660f827d-c694-49d5-9af0-1458f46f2ffb\") " pod="openshift-insights/insights-runtime-extractor-gnxgr" Jul 06 19:28:16.498747 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:16.498703 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/660f827d-c694-49d5-9af0-1458f46f2ffb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnxgr\" (UID: \"660f827d-c694-49d5-9af0-1458f46f2ffb\") " pod="openshift-insights/insights-runtime-extractor-gnxgr" Jul 06 19:28:16.599873 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:16.599837 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/660f827d-c694-49d5-9af0-1458f46f2ffb-crio-socket\") pod \"insights-runtime-extractor-gnxgr\" (UID: \"660f827d-c694-49d5-9af0-1458f46f2ffb\") " pod="openshift-insights/insights-runtime-extractor-gnxgr" Jul 06 19:28:16.599873 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:16.599876 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/660f827d-c694-49d5-9af0-1458f46f2ffb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gnxgr\" (UID: \"660f827d-c694-49d5-9af0-1458f46f2ffb\") " pod="openshift-insights/insights-runtime-extractor-gnxgr" Jul 06 19:28:16.600089 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:16.599902 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/660f827d-c694-49d5-9af0-1458f46f2ffb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnxgr\" (UID: \"660f827d-c694-49d5-9af0-1458f46f2ffb\") " pod="openshift-insights/insights-runtime-extractor-gnxgr" Jul 06 19:28:16.600089 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:16.599977 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/660f827d-c694-49d5-9af0-1458f46f2ffb-crio-socket\") pod \"insights-runtime-extractor-gnxgr\" (UID: \"660f827d-c694-49d5-9af0-1458f46f2ffb\") " pod="openshift-insights/insights-runtime-extractor-gnxgr" Jul 06 19:28:16.600089 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:16.600007 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/660f827d-c694-49d5-9af0-1458f46f2ffb-data-volume\") pod \"insights-runtime-extractor-gnxgr\" (UID: \"660f827d-c694-49d5-9af0-1458f46f2ffb\") " pod="openshift-insights/insights-runtime-extractor-gnxgr" Jul 06 19:28:16.600089 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:16.600068 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzcb4\" (UniqueName: \"kubernetes.io/projected/660f827d-c694-49d5-9af0-1458f46f2ffb-kube-api-access-zzcb4\") pod \"insights-runtime-extractor-gnxgr\" (UID: \"660f827d-c694-49d5-9af0-1458f46f2ffb\") " pod="openshift-insights/insights-runtime-extractor-gnxgr" Jul 06 19:28:16.600301 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:16.600013 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:28:16.600301 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:16.600043 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:28:16.600301 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:16.600232 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/660f827d-c694-49d5-9af0-1458f46f2ffb-insights-runtime-extractor-tls podName:660f827d-c694-49d5-9af0-1458f46f2ffb nodeName:}" failed. No retries permitted until 2026-07-06 19:28:17.100211353 +0000 UTC m=+105.920300044 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/660f827d-c694-49d5-9af0-1458f46f2ffb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gnxgr" (UID: "660f827d-c694-49d5-9af0-1458f46f2ffb") : secret "insights-runtime-extractor-tls" not found Jul 06 19:28:16.600301 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:16.600259 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/660f827d-c694-49d5-9af0-1458f46f2ffb-kube-rbac-proxy-cm podName:660f827d-c694-49d5-9af0-1458f46f2ffb nodeName:}" failed. No retries permitted until 2026-07-06 19:28:17.100245848 +0000 UTC m=+105.920334537 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/660f827d-c694-49d5-9af0-1458f46f2ffb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gnxgr" (UID: "660f827d-c694-49d5-9af0-1458f46f2ffb") : configmap "kube-rbac-proxy" not found Jul 06 19:28:16.600510 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:16.600334 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/660f827d-c694-49d5-9af0-1458f46f2ffb-data-volume\") pod \"insights-runtime-extractor-gnxgr\" (UID: \"660f827d-c694-49d5-9af0-1458f46f2ffb\") " pod="openshift-insights/insights-runtime-extractor-gnxgr" Jul 06 19:28:16.619598 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:16.619576 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzcb4\" (UniqueName: \"kubernetes.io/projected/660f827d-c694-49d5-9af0-1458f46f2ffb-kube-api-access-zzcb4\") pod \"insights-runtime-extractor-gnxgr\" (UID: \"660f827d-c694-49d5-9af0-1458f46f2ffb\") " pod="openshift-insights/insights-runtime-extractor-gnxgr" Jul 06 19:28:17.104038 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:17.103962 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/660f827d-c694-49d5-9af0-1458f46f2ffb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gnxgr\" (UID: \"660f827d-c694-49d5-9af0-1458f46f2ffb\") " pod="openshift-insights/insights-runtime-extractor-gnxgr" Jul 06 19:28:17.104038 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:17.103998 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/660f827d-c694-49d5-9af0-1458f46f2ffb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnxgr\" (UID: \"660f827d-c694-49d5-9af0-1458f46f2ffb\") " pod="openshift-insights/insights-runtime-extractor-gnxgr" Jul 06 19:28:17.104281 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:17.104092 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:28:17.104281 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:17.104143 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:28:17.104281 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:17.104162 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/660f827d-c694-49d5-9af0-1458f46f2ffb-kube-rbac-proxy-cm podName:660f827d-c694-49d5-9af0-1458f46f2ffb nodeName:}" failed. No retries permitted until 2026-07-06 19:28:18.104143727 +0000 UTC m=+106.924232427 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/660f827d-c694-49d5-9af0-1458f46f2ffb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gnxgr" (UID: "660f827d-c694-49d5-9af0-1458f46f2ffb") : configmap "kube-rbac-proxy" not found Jul 06 19:28:17.104281 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:17.104203 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/660f827d-c694-49d5-9af0-1458f46f2ffb-insights-runtime-extractor-tls podName:660f827d-c694-49d5-9af0-1458f46f2ffb nodeName:}" failed. No retries permitted until 2026-07-06 19:28:18.104172482 +0000 UTC m=+106.924261171 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/660f827d-c694-49d5-9af0-1458f46f2ffb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gnxgr" (UID: "660f827d-c694-49d5-9af0-1458f46f2ffb") : secret "insights-runtime-extractor-tls" not found Jul 06 19:28:17.172064 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:17.172036 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-b8955" event={"ID":"70caece2-9e4c-4518-9a0d-b36a8cc17d50","Type":"ContainerStarted","Data":"2a44000cddb19b6859b4a2a2f4f4b49ae975e3bfe4a12142c28cfd53b71fa82b"} Jul 06 19:28:17.204133 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:17.204092 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-b8955" podStartSLOduration=2.036499916 podStartE2EDuration="3.204078752s" totalImagesPullingTime="1.167578836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:28:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:28:17.200698615 +0000 UTC m=+106.020787324" watchObservedRunningTime="2026-07-06 19:28:17.204078752 +0000 UTC m=+106.024167462" Jul 06 19:28:18.110828 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:18.110794 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/660f827d-c694-49d5-9af0-1458f46f2ffb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gnxgr\" (UID: \"660f827d-c694-49d5-9af0-1458f46f2ffb\") " pod="openshift-insights/insights-runtime-extractor-gnxgr" Jul 06 19:28:18.110828 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:18.110832 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/660f827d-c694-49d5-9af0-1458f46f2ffb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnxgr\" (UID: \"660f827d-c694-49d5-9af0-1458f46f2ffb\") " pod="openshift-insights/insights-runtime-extractor-gnxgr" Jul 06 19:28:18.111318 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:18.110924 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:28:18.111318 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:18.110936 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:28:18.111318 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:18.110989 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/660f827d-c694-49d5-9af0-1458f46f2ffb-insights-runtime-extractor-tls podName:660f827d-c694-49d5-9af0-1458f46f2ffb nodeName:}" failed. No retries permitted until 2026-07-06 19:28:20.110976294 +0000 UTC m=+108.931064983 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/660f827d-c694-49d5-9af0-1458f46f2ffb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gnxgr" (UID: "660f827d-c694-49d5-9af0-1458f46f2ffb") : secret "insights-runtime-extractor-tls" not found Jul 06 19:28:18.111318 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:18.111004 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/660f827d-c694-49d5-9af0-1458f46f2ffb-kube-rbac-proxy-cm podName:660f827d-c694-49d5-9af0-1458f46f2ffb nodeName:}" failed. No retries permitted until 2026-07-06 19:28:20.11099737 +0000 UTC m=+108.931086059 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/660f827d-c694-49d5-9af0-1458f46f2ffb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gnxgr" (UID: "660f827d-c694-49d5-9af0-1458f46f2ffb") : configmap "kube-rbac-proxy" not found Jul 06 19:28:18.242857 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:18.242834 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2xwfj_9bd815c1-1206-431e-9d76-ba36f79833ac/dns-node-resolver/0.log" Jul 06 19:28:18.513938 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:18.513868 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/55480535-e86a-460e-b1bf-5c8da9c2ffa3-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-kxqkc\" (UID: \"55480535-e86a-460e-b1bf-5c8da9c2ffa3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-kxqkc" Jul 06 19:28:18.514054 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:18.514027 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 19:28:18.514105 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:18.514092 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55480535-e86a-460e-b1bf-5c8da9c2ffa3-samples-operator-tls podName:55480535-e86a-460e-b1bf-5c8da9c2ffa3 nodeName:}" failed. No retries permitted until 2026-07-06 19:28:22.514074714 +0000 UTC m=+111.334163421 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/55480535-e86a-460e-b1bf-5c8da9c2ffa3-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-kxqkc" (UID: "55480535-e86a-460e-b1bf-5c8da9c2ffa3") : secret "samples-operator-tls" not found Jul 06 19:28:19.443376 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:19.443341 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q2tnp_863c8526-b822-48d4-8881-819331146578/node-ca/0.log" Jul 06 19:28:19.847340 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:19.847308 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-96bb8_d0f9bf38-0857-458d-9962-51a2502b2a20/serve-healthcheck-canary/0.log" Jul 06 19:28:20.124371 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:20.124297 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/660f827d-c694-49d5-9af0-1458f46f2ffb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnxgr\" (UID: \"660f827d-c694-49d5-9af0-1458f46f2ffb\") " pod="openshift-insights/insights-runtime-extractor-gnxgr" Jul 06 19:28:20.124503 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:20.124393 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/660f827d-c694-49d5-9af0-1458f46f2ffb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gnxgr\" (UID: \"660f827d-c694-49d5-9af0-1458f46f2ffb\") " pod="openshift-insights/insights-runtime-extractor-gnxgr" Jul 06 19:28:20.124503 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:20.124437 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:28:20.124503 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:20.124466 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:28:20.124503 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:20.124494 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/660f827d-c694-49d5-9af0-1458f46f2ffb-insights-runtime-extractor-tls podName:660f827d-c694-49d5-9af0-1458f46f2ffb nodeName:}" failed. No retries permitted until 2026-07-06 19:28:24.124480426 +0000 UTC m=+112.944569120 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/660f827d-c694-49d5-9af0-1458f46f2ffb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gnxgr" (UID: "660f827d-c694-49d5-9af0-1458f46f2ffb") : secret "insights-runtime-extractor-tls" not found Jul 06 19:28:20.124630 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:20.124507 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/660f827d-c694-49d5-9af0-1458f46f2ffb-kube-rbac-proxy-cm podName:660f827d-c694-49d5-9af0-1458f46f2ffb nodeName:}" failed. No retries permitted until 2026-07-06 19:28:24.12450161 +0000 UTC m=+112.944590299 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/660f827d-c694-49d5-9af0-1458f46f2ffb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gnxgr" (UID: "660f827d-c694-49d5-9af0-1458f46f2ffb") : configmap "kube-rbac-proxy" not found Jul 06 19:28:20.527046 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:20.526970 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/677b6ba1-3252-4777-88fb-b4238918b699-service-ca-bundle\") pod \"router-default-6dcdb5648f-whl8g\" (UID: \"677b6ba1-3252-4777-88fb-b4238918b699\") " pod="openshift-ingress/router-default-6dcdb5648f-whl8g" Jul 06 19:28:20.527415 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:20.527068 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/677b6ba1-3252-4777-88fb-b4238918b699-metrics-certs\") pod \"router-default-6dcdb5648f-whl8g\" (UID: \"677b6ba1-3252-4777-88fb-b4238918b699\") " pod="openshift-ingress/router-default-6dcdb5648f-whl8g" Jul 06 19:28:20.527415 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:20.527145 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/677b6ba1-3252-4777-88fb-b4238918b699-service-ca-bundle podName:677b6ba1-3252-4777-88fb-b4238918b699 nodeName:}" failed. No retries permitted until 2026-07-06 19:28:28.527127249 +0000 UTC m=+117.347215942 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/677b6ba1-3252-4777-88fb-b4238918b699-service-ca-bundle") pod "router-default-6dcdb5648f-whl8g" (UID: "677b6ba1-3252-4777-88fb-b4238918b699") : configmap references non-existent config key: service-ca.crt Jul 06 19:28:20.527415 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:20.527163 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 19:28:20.527415 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:20.527236 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/677b6ba1-3252-4777-88fb-b4238918b699-metrics-certs podName:677b6ba1-3252-4777-88fb-b4238918b699 nodeName:}" failed. No retries permitted until 2026-07-06 19:28:28.527223443 +0000 UTC m=+117.347312132 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/677b6ba1-3252-4777-88fb-b4238918b699-metrics-certs") pod "router-default-6dcdb5648f-whl8g" (UID: "677b6ba1-3252-4777-88fb-b4238918b699") : secret "router-metrics-certs-default" not found Jul 06 19:28:22.541647 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:22.541612 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/55480535-e86a-460e-b1bf-5c8da9c2ffa3-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-kxqkc\" (UID: \"55480535-e86a-460e-b1bf-5c8da9c2ffa3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-kxqkc" Jul 06 19:28:22.542030 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:22.541752 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 19:28:22.542030 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:22.541821 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55480535-e86a-460e-b1bf-5c8da9c2ffa3-samples-operator-tls podName:55480535-e86a-460e-b1bf-5c8da9c2ffa3 nodeName:}" failed. No retries permitted until 2026-07-06 19:28:30.541805616 +0000 UTC m=+119.361894305 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/55480535-e86a-460e-b1bf-5c8da9c2ffa3-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-kxqkc" (UID: "55480535-e86a-460e-b1bf-5c8da9c2ffa3") : secret "samples-operator-tls" not found Jul 06 19:28:24.153255 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:24.153217 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/660f827d-c694-49d5-9af0-1458f46f2ffb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gnxgr\" (UID: \"660f827d-c694-49d5-9af0-1458f46f2ffb\") " pod="openshift-insights/insights-runtime-extractor-gnxgr" Jul 06 19:28:24.153255 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:24.153262 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/660f827d-c694-49d5-9af0-1458f46f2ffb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnxgr\" (UID: \"660f827d-c694-49d5-9af0-1458f46f2ffb\") " pod="openshift-insights/insights-runtime-extractor-gnxgr" Jul 06 19:28:24.153741 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:24.153333 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:28:24.153741 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:24.153387 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:28:24.153741 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:24.153399 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/660f827d-c694-49d5-9af0-1458f46f2ffb-kube-rbac-proxy-cm podName:660f827d-c694-49d5-9af0-1458f46f2ffb nodeName:}" failed. No retries permitted until 2026-07-06 19:28:32.153379529 +0000 UTC m=+120.973468222 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/660f827d-c694-49d5-9af0-1458f46f2ffb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gnxgr" (UID: "660f827d-c694-49d5-9af0-1458f46f2ffb") : configmap "kube-rbac-proxy" not found Jul 06 19:28:24.153741 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:24.153446 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/660f827d-c694-49d5-9af0-1458f46f2ffb-insights-runtime-extractor-tls podName:660f827d-c694-49d5-9af0-1458f46f2ffb nodeName:}" failed. No retries permitted until 2026-07-06 19:28:32.153430506 +0000 UTC m=+120.973519233 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/660f827d-c694-49d5-9af0-1458f46f2ffb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gnxgr" (UID: "660f827d-c694-49d5-9af0-1458f46f2ffb") : secret "insights-runtime-extractor-tls" not found Jul 06 19:28:28.585047 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:28.585003 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/677b6ba1-3252-4777-88fb-b4238918b699-service-ca-bundle\") pod \"router-default-6dcdb5648f-whl8g\" (UID: \"677b6ba1-3252-4777-88fb-b4238918b699\") " pod="openshift-ingress/router-default-6dcdb5648f-whl8g" Jul 06 19:28:28.585587 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:28.585205 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/677b6ba1-3252-4777-88fb-b4238918b699-metrics-certs\") pod \"router-default-6dcdb5648f-whl8g\" (UID: \"677b6ba1-3252-4777-88fb-b4238918b699\") " pod="openshift-ingress/router-default-6dcdb5648f-whl8g" Jul 06 19:28:28.585587 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:28.585485 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 19:28:28.585701 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:28.585617 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/677b6ba1-3252-4777-88fb-b4238918b699-metrics-certs podName:677b6ba1-3252-4777-88fb-b4238918b699 nodeName:}" failed. No retries permitted until 2026-07-06 19:28:44.585582958 +0000 UTC m=+133.405671656 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/677b6ba1-3252-4777-88fb-b4238918b699-metrics-certs") pod "router-default-6dcdb5648f-whl8g" (UID: "677b6ba1-3252-4777-88fb-b4238918b699") : secret "router-metrics-certs-default" not found Jul 06 19:28:28.586203 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:28.586164 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/677b6ba1-3252-4777-88fb-b4238918b699-service-ca-bundle podName:677b6ba1-3252-4777-88fb-b4238918b699 nodeName:}" failed. No retries permitted until 2026-07-06 19:28:44.586146256 +0000 UTC m=+133.406234959 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/677b6ba1-3252-4777-88fb-b4238918b699-service-ca-bundle") pod "router-default-6dcdb5648f-whl8g" (UID: "677b6ba1-3252-4777-88fb-b4238918b699") : configmap references non-existent config key: service-ca.crt Jul 06 19:28:28.818677 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:28.818649 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-vkf2c"] Jul 06 19:28:28.820625 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:28.820610 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-vkf2c" Jul 06 19:28:28.823970 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:28.823946 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 06 19:28:28.823970 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:28.823972 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 06 19:28:28.825094 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:28.825078 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 06 19:28:28.825193 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:28.825172 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 06 19:28:28.825273 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:28.825176 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-7ct47\"" Jul 06 19:28:28.852779 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:28.852730 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-vkf2c"] Jul 06 19:28:28.887331 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:28.887310 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck88v\" (UniqueName: \"kubernetes.io/projected/58857db6-8922-43ac-9c06-1d18d2e01037-kube-api-access-ck88v\") pod \"service-ca-758f599c5-vkf2c\" (UID: \"58857db6-8922-43ac-9c06-1d18d2e01037\") " pod="openshift-service-ca/service-ca-758f599c5-vkf2c" Jul 06 19:28:28.887420 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:28.887342 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/58857db6-8922-43ac-9c06-1d18d2e01037-signing-cabundle\") pod \"service-ca-758f599c5-vkf2c\" (UID: \"58857db6-8922-43ac-9c06-1d18d2e01037\") " pod="openshift-service-ca/service-ca-758f599c5-vkf2c" Jul 06 19:28:28.887420 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:28.887374 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/58857db6-8922-43ac-9c06-1d18d2e01037-signing-key\") pod \"service-ca-758f599c5-vkf2c\" (UID: \"58857db6-8922-43ac-9c06-1d18d2e01037\") " pod="openshift-service-ca/service-ca-758f599c5-vkf2c" Jul 06 19:28:28.987899 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:28.987881 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck88v\" (UniqueName: \"kubernetes.io/projected/58857db6-8922-43ac-9c06-1d18d2e01037-kube-api-access-ck88v\") pod \"service-ca-758f599c5-vkf2c\" (UID: \"58857db6-8922-43ac-9c06-1d18d2e01037\") " pod="openshift-service-ca/service-ca-758f599c5-vkf2c" Jul 06 19:28:28.987994 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:28.987910 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/58857db6-8922-43ac-9c06-1d18d2e01037-signing-cabundle\") pod \"service-ca-758f599c5-vkf2c\" (UID: \"58857db6-8922-43ac-9c06-1d18d2e01037\") " pod="openshift-service-ca/service-ca-758f599c5-vkf2c" Jul 06 19:28:28.987994 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:28.987930 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/58857db6-8922-43ac-9c06-1d18d2e01037-signing-key\") pod \"service-ca-758f599c5-vkf2c\" (UID: \"58857db6-8922-43ac-9c06-1d18d2e01037\") " pod="openshift-service-ca/service-ca-758f599c5-vkf2c" Jul 06 19:28:28.988487 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:28.988467 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/58857db6-8922-43ac-9c06-1d18d2e01037-signing-cabundle\") pod \"service-ca-758f599c5-vkf2c\" (UID: \"58857db6-8922-43ac-9c06-1d18d2e01037\") " pod="openshift-service-ca/service-ca-758f599c5-vkf2c" Jul 06 19:28:28.990043 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:28.990017 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/58857db6-8922-43ac-9c06-1d18d2e01037-signing-key\") pod \"service-ca-758f599c5-vkf2c\" (UID: \"58857db6-8922-43ac-9c06-1d18d2e01037\") " pod="openshift-service-ca/service-ca-758f599c5-vkf2c" Jul 06 19:28:28.997586 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:28.997568 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck88v\" (UniqueName: \"kubernetes.io/projected/58857db6-8922-43ac-9c06-1d18d2e01037-kube-api-access-ck88v\") pod \"service-ca-758f599c5-vkf2c\" (UID: \"58857db6-8922-43ac-9c06-1d18d2e01037\") " pod="openshift-service-ca/service-ca-758f599c5-vkf2c" Jul 06 19:28:29.129146 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:29.129094 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-vkf2c" Jul 06 19:28:29.256380 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:29.256349 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-vkf2c"] Jul 06 19:28:29.259989 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:28:29.259953 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58857db6_8922_43ac_9c06_1d18d2e01037.slice/crio-6e489030c79be2085ef70aff5d9e7b902f25e0643b75d95c058b25e2fdcc2580 WatchSource:0}: Error finding container 6e489030c79be2085ef70aff5d9e7b902f25e0643b75d95c058b25e2fdcc2580: Status 404 returned error can't find the container with id 6e489030c79be2085ef70aff5d9e7b902f25e0643b75d95c058b25e2fdcc2580 Jul 06 19:28:30.205767 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:30.205732 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-vkf2c" event={"ID":"58857db6-8922-43ac-9c06-1d18d2e01037","Type":"ContainerStarted","Data":"6e489030c79be2085ef70aff5d9e7b902f25e0643b75d95c058b25e2fdcc2580"} Jul 06 19:28:30.602837 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:30.602804 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/55480535-e86a-460e-b1bf-5c8da9c2ffa3-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-kxqkc\" (UID: \"55480535-e86a-460e-b1bf-5c8da9c2ffa3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-kxqkc" Jul 06 19:28:30.603047 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:30.602910 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 19:28:30.603047 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:30.602977 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55480535-e86a-460e-b1bf-5c8da9c2ffa3-samples-operator-tls podName:55480535-e86a-460e-b1bf-5c8da9c2ffa3 nodeName:}" failed. No retries permitted until 2026-07-06 19:28:46.602957897 +0000 UTC m=+135.423046599 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/55480535-e86a-460e-b1bf-5c8da9c2ffa3-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-kxqkc" (UID: "55480535-e86a-460e-b1bf-5c8da9c2ffa3") : secret "samples-operator-tls" not found Jul 06 19:28:32.212755 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:32.212724 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-vkf2c" event={"ID":"58857db6-8922-43ac-9c06-1d18d2e01037","Type":"ContainerStarted","Data":"32881ed677869fedceef76b614485e54c47d8ff09482985fdd3480a6f8740ad3"} Jul 06 19:28:32.215207 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:32.215169 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/660f827d-c694-49d5-9af0-1458f46f2ffb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnxgr\" (UID: \"660f827d-c694-49d5-9af0-1458f46f2ffb\") " pod="openshift-insights/insights-runtime-extractor-gnxgr" Jul 06 19:28:32.215284 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:32.215230 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/660f827d-c694-49d5-9af0-1458f46f2ffb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gnxgr\" (UID: \"660f827d-c694-49d5-9af0-1458f46f2ffb\") " pod="openshift-insights/insights-runtime-extractor-gnxgr" Jul 06 19:28:32.215321 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:32.215282 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:28:32.215364 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:32.215355 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/660f827d-c694-49d5-9af0-1458f46f2ffb-insights-runtime-extractor-tls podName:660f827d-c694-49d5-9af0-1458f46f2ffb nodeName:}" failed. No retries permitted until 2026-07-06 19:28:48.215338157 +0000 UTC m=+137.035426859 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/660f827d-c694-49d5-9af0-1458f46f2ffb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gnxgr" (UID: "660f827d-c694-49d5-9af0-1458f46f2ffb") : secret "insights-runtime-extractor-tls" not found Jul 06 19:28:32.215687 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:32.215669 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/660f827d-c694-49d5-9af0-1458f46f2ffb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gnxgr\" (UID: \"660f827d-c694-49d5-9af0-1458f46f2ffb\") " pod="openshift-insights/insights-runtime-extractor-gnxgr" Jul 06 19:28:32.233954 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:32.233910 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-vkf2c" podStartSLOduration=1.982629204 podStartE2EDuration="4.233897366s" totalImagesPullingTime="2.251268162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:28:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:28:32.233008188 +0000 UTC m=+121.053096910" watchObservedRunningTime="2026-07-06 19:28:32.233897366 +0000 UTC m=+121.053986076" Jul 06 19:28:41.385351 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:41.385318 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2f29af8-1539-48c2-a77e-3b7691f5cea8-metrics-certs\") pod \"network-metrics-daemon-ptml7\" (UID: \"c2f29af8-1539-48c2-a77e-3b7691f5cea8\") " pod="openshift-multus/network-metrics-daemon-ptml7" Jul 06 19:28:41.387728 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:41.387704 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2f29af8-1539-48c2-a77e-3b7691f5cea8-metrics-certs\") pod \"network-metrics-daemon-ptml7\" (UID: \"c2f29af8-1539-48c2-a77e-3b7691f5cea8\") " pod="openshift-multus/network-metrics-daemon-ptml7" Jul 06 19:28:41.615616 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:41.615589 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-t5qdm\"" Jul 06 19:28:41.623538 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:41.623510 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ptml7" Jul 06 19:28:41.752609 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:41.752581 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ptml7"] Jul 06 19:28:41.754385 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:28:41.754357 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2f29af8_1539_48c2_a77e_3b7691f5cea8.slice/crio-30e916e863352e11d47c7a9bf415bcaa428b12291a8f0cdc8e36e3a8ea5836f8 WatchSource:0}: Error finding container 30e916e863352e11d47c7a9bf415bcaa428b12291a8f0cdc8e36e3a8ea5836f8: Status 404 returned error can't find the container with id 30e916e863352e11d47c7a9bf415bcaa428b12291a8f0cdc8e36e3a8ea5836f8 Jul 06 19:28:42.243674 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:42.243640 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ptml7" event={"ID":"c2f29af8-1539-48c2-a77e-3b7691f5cea8","Type":"ContainerStarted","Data":"30e916e863352e11d47c7a9bf415bcaa428b12291a8f0cdc8e36e3a8ea5836f8"} Jul 06 19:28:43.248973 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:43.248902 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ptml7" event={"ID":"c2f29af8-1539-48c2-a77e-3b7691f5cea8","Type":"ContainerStarted","Data":"ca6b7fdfadc9ebd081931dc5d3155e4ec62d7236f7a430d07f1d6f3e4b7fdcfc"} Jul 06 19:28:43.248973 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:43.248954 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ptml7" event={"ID":"c2f29af8-1539-48c2-a77e-3b7691f5cea8","Type":"ContainerStarted","Data":"fdc43705e833bdd06bf3c18c52fa8ce228a3a79569424e5e2049c224128395e0"} Jul 06 19:28:43.267026 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:43.266980 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ptml7" podStartSLOduration=131.178238358 podStartE2EDuration="2m12.266966366s" totalImagesPullingTime="1.088728008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:28:43.266347612 +0000 UTC m=+132.086436314" watchObservedRunningTime="2026-07-06 19:28:43.266966366 +0000 UTC m=+132.087055077" Jul 06 19:28:44.607216 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:44.607169 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/677b6ba1-3252-4777-88fb-b4238918b699-metrics-certs\") pod \"router-default-6dcdb5648f-whl8g\" (UID: \"677b6ba1-3252-4777-88fb-b4238918b699\") " pod="openshift-ingress/router-default-6dcdb5648f-whl8g" Jul 06 19:28:44.607216 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:44.607220 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/677b6ba1-3252-4777-88fb-b4238918b699-service-ca-bundle\") pod \"router-default-6dcdb5648f-whl8g\" (UID: \"677b6ba1-3252-4777-88fb-b4238918b699\") " pod="openshift-ingress/router-default-6dcdb5648f-whl8g" Jul 06 19:28:44.607736 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:44.607718 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/677b6ba1-3252-4777-88fb-b4238918b699-service-ca-bundle\") pod \"router-default-6dcdb5648f-whl8g\" (UID: \"677b6ba1-3252-4777-88fb-b4238918b699\") " pod="openshift-ingress/router-default-6dcdb5648f-whl8g" Jul 06 19:28:44.609371 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:44.609351 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/677b6ba1-3252-4777-88fb-b4238918b699-metrics-certs\") pod \"router-default-6dcdb5648f-whl8g\" (UID: \"677b6ba1-3252-4777-88fb-b4238918b699\") " pod="openshift-ingress/router-default-6dcdb5648f-whl8g" Jul 06 19:28:44.663157 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:44.663131 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-jjtj8\"" Jul 06 19:28:44.670472 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:44.670457 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6dcdb5648f-whl8g" Jul 06 19:28:44.791669 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:44.791641 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6dcdb5648f-whl8g"] Jul 06 19:28:44.794368 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:28:44.794335 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod677b6ba1_3252_4777_88fb_b4238918b699.slice/crio-1969656122f6fe05bb161c1e07c9421cc6031bd5ce3254f8a0dd135fb3609960 WatchSource:0}: Error finding container 1969656122f6fe05bb161c1e07c9421cc6031bd5ce3254f8a0dd135fb3609960: Status 404 returned error can't find the container with id 1969656122f6fe05bb161c1e07c9421cc6031bd5ce3254f8a0dd135fb3609960 Jul 06 19:28:45.257462 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:45.257432 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6dcdb5648f-whl8g" event={"ID":"677b6ba1-3252-4777-88fb-b4238918b699","Type":"ContainerStarted","Data":"92b5fecea030733ecd16f4d4550a84730d5d29b0622f1626fbe1a159acca7b04"} Jul 06 19:28:45.257597 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:45.257481 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6dcdb5648f-whl8g" event={"ID":"677b6ba1-3252-4777-88fb-b4238918b699","Type":"ContainerStarted","Data":"1969656122f6fe05bb161c1e07c9421cc6031bd5ce3254f8a0dd135fb3609960"} Jul 06 19:28:45.281359 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:45.281318 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6dcdb5648f-whl8g" podStartSLOduration=33.281301981 podStartE2EDuration="33.281301981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:28:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:28:45.280417506 +0000 UTC m=+134.100506240" watchObservedRunningTime="2026-07-06 19:28:45.281301981 +0000 UTC m=+134.101390691" Jul 06 19:28:45.670884 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:45.670850 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6dcdb5648f-whl8g" Jul 06 19:28:45.673303 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:45.673278 2575 patch_prober.go:28] interesting pod/router-default-6dcdb5648f-whl8g container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 06 19:28:45.673303 ip-10-0-135-23 kubenswrapper[2575]: [-]has-synced failed: reason withheld Jul 06 19:28:45.673303 ip-10-0-135-23 kubenswrapper[2575]: [+]process-running ok Jul 06 19:28:45.673303 ip-10-0-135-23 kubenswrapper[2575]: healthz check failed Jul 06 19:28:45.673501 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:45.673326 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-6dcdb5648f-whl8g" podUID="677b6ba1-3252-4777-88fb-b4238918b699" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 06 19:28:46.620977 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:46.620938 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/55480535-e86a-460e-b1bf-5c8da9c2ffa3-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-kxqkc\" (UID: \"55480535-e86a-460e-b1bf-5c8da9c2ffa3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-kxqkc" Jul 06 19:28:46.623091 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:46.623069 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/55480535-e86a-460e-b1bf-5c8da9c2ffa3-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-kxqkc\" (UID: \"55480535-e86a-460e-b1bf-5c8da9c2ffa3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-kxqkc" Jul 06 19:28:46.672687 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:46.672668 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6dcdb5648f-whl8g" Jul 06 19:28:46.897094 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:46.893802 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-h45tb\"" Jul 06 19:28:46.900929 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:46.900912 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-kxqkc" Jul 06 19:28:47.043064 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:47.043035 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-kxqkc"] Jul 06 19:28:47.265842 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:47.265766 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-kxqkc" event={"ID":"55480535-e86a-460e-b1bf-5c8da9c2ffa3","Type":"ContainerStarted","Data":"fa23dca1a40f04c5a05f017437d82fdeac8a12b96a493872863f15d97941ec62"} Jul 06 19:28:47.266107 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:47.266088 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6dcdb5648f-whl8g" Jul 06 19:28:47.267211 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:47.267194 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6dcdb5648f-whl8g" Jul 06 19:28:48.231990 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:48.231959 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/660f827d-c694-49d5-9af0-1458f46f2ffb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnxgr\" (UID: \"660f827d-c694-49d5-9af0-1458f46f2ffb\") " pod="openshift-insights/insights-runtime-extractor-gnxgr" Jul 06 19:28:48.234091 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:48.234066 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/660f827d-c694-49d5-9af0-1458f46f2ffb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnxgr\" (UID: \"660f827d-c694-49d5-9af0-1458f46f2ffb\") " pod="openshift-insights/insights-runtime-extractor-gnxgr" Jul 06 19:28:48.280285 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:48.280265 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k82dj\"" Jul 06 19:28:48.288407 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:48.288392 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gnxgr" Jul 06 19:28:48.447778 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:48.447749 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gnxgr"] Jul 06 19:28:48.450417 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:28:48.450384 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod660f827d_c694_49d5_9af0_1458f46f2ffb.slice/crio-414f1a5b7df20aaf223cac145c277f27e7d87b7f2dd50642ed8c20625b7bd053 WatchSource:0}: Error finding container 414f1a5b7df20aaf223cac145c277f27e7d87b7f2dd50642ed8c20625b7bd053: Status 404 returned error can't find the container with id 414f1a5b7df20aaf223cac145c277f27e7d87b7f2dd50642ed8c20625b7bd053 Jul 06 19:28:49.274787 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:49.274671 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-kxqkc" event={"ID":"55480535-e86a-460e-b1bf-5c8da9c2ffa3","Type":"ContainerStarted","Data":"00e9d98d366c0c5dba7e683e53556137a1a1b85189f047c229622896b60410c1"} Jul 06 19:28:49.274787 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:49.274740 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-kxqkc" event={"ID":"55480535-e86a-460e-b1bf-5c8da9c2ffa3","Type":"ContainerStarted","Data":"40670ea7c5b09e5211330acabd7daea7f6be1206fc5232fca9d1d22cc327ba47"} Jul 06 19:28:49.277817 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:49.277746 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gnxgr" event={"ID":"660f827d-c694-49d5-9af0-1458f46f2ffb","Type":"ContainerStarted","Data":"414f1a5b7df20aaf223cac145c277f27e7d87b7f2dd50642ed8c20625b7bd053"} Jul 06 19:28:49.304772 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:49.304717 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-kxqkc" podStartSLOduration=33.388445611 podStartE2EDuration="35.30469859s" totalImagesPullingTime="1.916252979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:28:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:28:49.303065998 +0000 UTC m=+138.123154707" watchObservedRunningTime="2026-07-06 19:28:49.30469859 +0000 UTC m=+138.124787303" Jul 06 19:28:52.290587 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:52.290552 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gnxgr" event={"ID":"660f827d-c694-49d5-9af0-1458f46f2ffb","Type":"ContainerStarted","Data":"d205c76e7ca0df7c1edd1c90d0ea38c4f32027ae8eb0067ad43b436820a10b9b"} Jul 06 19:28:52.841676 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:52.841653 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-rsg5c"] Jul 06 19:28:52.844721 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:52.844703 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-rsg5c" Jul 06 19:28:52.851883 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:52.851865 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bf496c95-7q6sp"] Jul 06 19:28:52.854768 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:52.854749 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bf496c95-7q6sp" Jul 06 19:28:52.879506 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:52.879491 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 06 19:28:52.879599 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:52.879491 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 06 19:28:52.879599 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:52.879560 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 06 19:28:52.879710 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:52.879602 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-rjjgx\"" Jul 06 19:28:52.879710 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:52.879599 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 06 19:28:52.880438 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:52.880426 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 06 19:28:52.880594 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:52.880579 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 06 19:28:52.880894 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:52.880879 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rk2kv\"" Jul 06 19:28:52.880932 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:52.880880 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 06 19:28:52.894419 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:52.894400 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-rsg5c"] Jul 06 19:28:52.901205 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:52.901166 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bf496c95-7q6sp"] Jul 06 19:28:52.965433 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:52.965410 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b07f9de-449f-4154-b0ee-2d06d6a93313-console-config\") pod \"console-7bf496c95-7q6sp\" (UID: \"6b07f9de-449f-4154-b0ee-2d06d6a93313\") " pod="openshift-console/console-7bf496c95-7q6sp" Jul 06 19:28:52.965553 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:52.965448 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b07f9de-449f-4154-b0ee-2d06d6a93313-console-oauth-config\") pod \"console-7bf496c95-7q6sp\" (UID: \"6b07f9de-449f-4154-b0ee-2d06d6a93313\") " pod="openshift-console/console-7bf496c95-7q6sp" Jul 06 19:28:52.965553 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:52.965469 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b07f9de-449f-4154-b0ee-2d06d6a93313-oauth-serving-cert\") pod \"console-7bf496c95-7q6sp\" (UID: \"6b07f9de-449f-4154-b0ee-2d06d6a93313\") " pod="openshift-console/console-7bf496c95-7q6sp" Jul 06 19:28:52.965553 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:52.965542 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qpdl\" (UniqueName: \"kubernetes.io/projected/6b07f9de-449f-4154-b0ee-2d06d6a93313-kube-api-access-8qpdl\") pod \"console-7bf496c95-7q6sp\" (UID: \"6b07f9de-449f-4154-b0ee-2d06d6a93313\") " pod="openshift-console/console-7bf496c95-7q6sp" Jul 06 19:28:52.965711 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:52.965573 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phkfg\" (UniqueName: \"kubernetes.io/projected/d3f27b98-edb6-4774-a5ea-330b55f2e0ec-kube-api-access-phkfg\") pod \"downloads-685bfb5f64-rsg5c\" (UID: \"d3f27b98-edb6-4774-a5ea-330b55f2e0ec\") " pod="openshift-console/downloads-685bfb5f64-rsg5c" Jul 06 19:28:52.965711 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:52.965624 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b07f9de-449f-4154-b0ee-2d06d6a93313-service-ca\") pod \"console-7bf496c95-7q6sp\" (UID: \"6b07f9de-449f-4154-b0ee-2d06d6a93313\") " pod="openshift-console/console-7bf496c95-7q6sp" Jul 06 19:28:52.965711 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:52.965656 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b07f9de-449f-4154-b0ee-2d06d6a93313-console-serving-cert\") pod \"console-7bf496c95-7q6sp\" (UID: \"6b07f9de-449f-4154-b0ee-2d06d6a93313\") " pod="openshift-console/console-7bf496c95-7q6sp" Jul 06 19:28:53.067000 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:53.066942 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qpdl\" (UniqueName: \"kubernetes.io/projected/6b07f9de-449f-4154-b0ee-2d06d6a93313-kube-api-access-8qpdl\") pod \"console-7bf496c95-7q6sp\" (UID: \"6b07f9de-449f-4154-b0ee-2d06d6a93313\") " pod="openshift-console/console-7bf496c95-7q6sp" Jul 06 19:28:53.067000 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:53.066971 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phkfg\" (UniqueName: \"kubernetes.io/projected/d3f27b98-edb6-4774-a5ea-330b55f2e0ec-kube-api-access-phkfg\") pod \"downloads-685bfb5f64-rsg5c\" (UID: \"d3f27b98-edb6-4774-a5ea-330b55f2e0ec\") " pod="openshift-console/downloads-685bfb5f64-rsg5c" Jul 06 19:28:53.067000 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:53.066991 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b07f9de-449f-4154-b0ee-2d06d6a93313-service-ca\") pod \"console-7bf496c95-7q6sp\" (UID: \"6b07f9de-449f-4154-b0ee-2d06d6a93313\") " pod="openshift-console/console-7bf496c95-7q6sp" Jul 06 19:28:53.067236 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:53.067013 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b07f9de-449f-4154-b0ee-2d06d6a93313-console-serving-cert\") pod \"console-7bf496c95-7q6sp\" (UID: \"6b07f9de-449f-4154-b0ee-2d06d6a93313\") " pod="openshift-console/console-7bf496c95-7q6sp" Jul 06 19:28:53.067236 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:53.067046 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b07f9de-449f-4154-b0ee-2d06d6a93313-console-config\") pod \"console-7bf496c95-7q6sp\" (UID: \"6b07f9de-449f-4154-b0ee-2d06d6a93313\") " pod="openshift-console/console-7bf496c95-7q6sp" Jul 06 19:28:53.067236 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:53.067096 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b07f9de-449f-4154-b0ee-2d06d6a93313-console-oauth-config\") pod \"console-7bf496c95-7q6sp\" (UID: \"6b07f9de-449f-4154-b0ee-2d06d6a93313\") " pod="openshift-console/console-7bf496c95-7q6sp" Jul 06 19:28:53.067236 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:53.067123 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b07f9de-449f-4154-b0ee-2d06d6a93313-oauth-serving-cert\") pod \"console-7bf496c95-7q6sp\" (UID: \"6b07f9de-449f-4154-b0ee-2d06d6a93313\") " pod="openshift-console/console-7bf496c95-7q6sp" Jul 06 19:28:53.067742 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:53.067705 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b07f9de-449f-4154-b0ee-2d06d6a93313-oauth-serving-cert\") pod \"console-7bf496c95-7q6sp\" (UID: \"6b07f9de-449f-4154-b0ee-2d06d6a93313\") " pod="openshift-console/console-7bf496c95-7q6sp" Jul 06 19:28:53.067742 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:53.067710 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b07f9de-449f-4154-b0ee-2d06d6a93313-service-ca\") pod \"console-7bf496c95-7q6sp\" (UID: \"6b07f9de-449f-4154-b0ee-2d06d6a93313\") " pod="openshift-console/console-7bf496c95-7q6sp" Jul 06 19:28:53.067873 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:53.067839 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b07f9de-449f-4154-b0ee-2d06d6a93313-console-config\") pod \"console-7bf496c95-7q6sp\" (UID: \"6b07f9de-449f-4154-b0ee-2d06d6a93313\") " pod="openshift-console/console-7bf496c95-7q6sp" Jul 06 19:28:53.069338 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:53.069314 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b07f9de-449f-4154-b0ee-2d06d6a93313-console-serving-cert\") pod \"console-7bf496c95-7q6sp\" (UID: \"6b07f9de-449f-4154-b0ee-2d06d6a93313\") " pod="openshift-console/console-7bf496c95-7q6sp" Jul 06 19:28:53.069422 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:53.069404 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b07f9de-449f-4154-b0ee-2d06d6a93313-console-oauth-config\") pod \"console-7bf496c95-7q6sp\" (UID: \"6b07f9de-449f-4154-b0ee-2d06d6a93313\") " pod="openshift-console/console-7bf496c95-7q6sp" Jul 06 19:28:53.096759 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:53.096735 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qpdl\" (UniqueName: \"kubernetes.io/projected/6b07f9de-449f-4154-b0ee-2d06d6a93313-kube-api-access-8qpdl\") pod \"console-7bf496c95-7q6sp\" (UID: \"6b07f9de-449f-4154-b0ee-2d06d6a93313\") " pod="openshift-console/console-7bf496c95-7q6sp" Jul 06 19:28:53.097771 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:53.097748 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phkfg\" (UniqueName: \"kubernetes.io/projected/d3f27b98-edb6-4774-a5ea-330b55f2e0ec-kube-api-access-phkfg\") pod \"downloads-685bfb5f64-rsg5c\" (UID: \"d3f27b98-edb6-4774-a5ea-330b55f2e0ec\") " pod="openshift-console/downloads-685bfb5f64-rsg5c" Jul 06 19:28:53.152259 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:53.152240 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-rsg5c" Jul 06 19:28:53.163063 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:53.163046 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bf496c95-7q6sp" Jul 06 19:28:53.295760 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:53.295731 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gnxgr" event={"ID":"660f827d-c694-49d5-9af0-1458f46f2ffb","Type":"ContainerStarted","Data":"a73c08658aad952cd6ccf9f049428a6c56fdb3511536d9051b9dee2a21cd4234"} Jul 06 19:28:53.302369 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:53.302347 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-rsg5c"] Jul 06 19:28:53.304865 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:28:53.304835 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3f27b98_edb6_4774_a5ea_330b55f2e0ec.slice/crio-f147e37df5b8c8e1fbf8926e9cc523d059f43b0c797aee4fbedcd09c75ae6ef9 WatchSource:0}: Error finding container f147e37df5b8c8e1fbf8926e9cc523d059f43b0c797aee4fbedcd09c75ae6ef9: Status 404 returned error can't find the container with id f147e37df5b8c8e1fbf8926e9cc523d059f43b0c797aee4fbedcd09c75ae6ef9 Jul 06 19:28:53.309124 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:53.309104 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bf496c95-7q6sp"] Jul 06 19:28:53.311596 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:28:53.311571 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b07f9de_449f_4154_b0ee_2d06d6a93313.slice/crio-dfae2d9f8c94d9323585a1b4139f2176cf1ad505f711a5a1c92e39672229c2cd WatchSource:0}: Error finding container dfae2d9f8c94d9323585a1b4139f2176cf1ad505f711a5a1c92e39672229c2cd: Status 404 returned error can't find the container with id dfae2d9f8c94d9323585a1b4139f2176cf1ad505f711a5a1c92e39672229c2cd Jul 06 19:28:54.301837 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:54.301796 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bf496c95-7q6sp" event={"ID":"6b07f9de-449f-4154-b0ee-2d06d6a93313","Type":"ContainerStarted","Data":"dfae2d9f8c94d9323585a1b4139f2176cf1ad505f711a5a1c92e39672229c2cd"} Jul 06 19:28:54.303328 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:54.303300 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-rsg5c" event={"ID":"d3f27b98-edb6-4774-a5ea-330b55f2e0ec","Type":"ContainerStarted","Data":"f147e37df5b8c8e1fbf8926e9cc523d059f43b0c797aee4fbedcd09c75ae6ef9"} Jul 06 19:28:56.593793 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:56.593741 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-tjmst"] Jul 06 19:28:56.602595 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:56.602511 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-tjmst" Jul 06 19:28:56.605515 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:56.605489 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 06 19:28:56.605618 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:56.605563 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 19:28:56.607139 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:56.606452 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 19:28:56.607139 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:56.606809 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-rrn2b\"" Jul 06 19:28:56.607139 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:56.606994 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 19:28:56.607380 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:56.607140 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 06 19:28:56.612358 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:56.612307 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-tjmst"] Jul 06 19:28:56.698604 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:56.698577 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6a2716f-443f-4b0a-98f4-c5a88c2785c7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-tjmst\" (UID: \"e6a2716f-443f-4b0a-98f4-c5a88c2785c7\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-tjmst" Jul 06 19:28:56.698747 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:56.698615 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e6a2716f-443f-4b0a-98f4-c5a88c2785c7-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-tjmst\" (UID: \"e6a2716f-443f-4b0a-98f4-c5a88c2785c7\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-tjmst" Jul 06 19:28:56.698747 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:56.698693 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6a2716f-443f-4b0a-98f4-c5a88c2785c7-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-tjmst\" (UID: \"e6a2716f-443f-4b0a-98f4-c5a88c2785c7\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-tjmst" Jul 06 19:28:56.698839 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:56.698770 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd8kx\" (UniqueName: \"kubernetes.io/projected/e6a2716f-443f-4b0a-98f4-c5a88c2785c7-kube-api-access-nd8kx\") pod \"prometheus-operator-567f7b8b7c-tjmst\" (UID: \"e6a2716f-443f-4b0a-98f4-c5a88c2785c7\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-tjmst" Jul 06 19:28:56.800010 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:56.799979 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd8kx\" (UniqueName: \"kubernetes.io/projected/e6a2716f-443f-4b0a-98f4-c5a88c2785c7-kube-api-access-nd8kx\") pod \"prometheus-operator-567f7b8b7c-tjmst\" (UID: \"e6a2716f-443f-4b0a-98f4-c5a88c2785c7\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-tjmst" Jul 06 19:28:56.800172 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:56.800151 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6a2716f-443f-4b0a-98f4-c5a88c2785c7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-tjmst\" (UID: \"e6a2716f-443f-4b0a-98f4-c5a88c2785c7\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-tjmst" Jul 06 19:28:56.800259 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:56.800208 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e6a2716f-443f-4b0a-98f4-c5a88c2785c7-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-tjmst\" (UID: \"e6a2716f-443f-4b0a-98f4-c5a88c2785c7\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-tjmst" Jul 06 19:28:56.800259 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:56.800248 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6a2716f-443f-4b0a-98f4-c5a88c2785c7-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-tjmst\" (UID: \"e6a2716f-443f-4b0a-98f4-c5a88c2785c7\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-tjmst" Jul 06 19:28:56.800423 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:56.800409 2575 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 06 19:28:56.800488 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:28:56.800475 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6a2716f-443f-4b0a-98f4-c5a88c2785c7-prometheus-operator-tls podName:e6a2716f-443f-4b0a-98f4-c5a88c2785c7 nodeName:}" failed. No retries permitted until 2026-07-06 19:28:57.300455528 +0000 UTC m=+146.120544223 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/e6a2716f-443f-4b0a-98f4-c5a88c2785c7-prometheus-operator-tls") pod "prometheus-operator-567f7b8b7c-tjmst" (UID: "e6a2716f-443f-4b0a-98f4-c5a88c2785c7") : secret "prometheus-operator-tls" not found Jul 06 19:28:56.801011 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:56.800987 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6a2716f-443f-4b0a-98f4-c5a88c2785c7-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-tjmst\" (UID: \"e6a2716f-443f-4b0a-98f4-c5a88c2785c7\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-tjmst" Jul 06 19:28:56.802908 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:56.802881 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6a2716f-443f-4b0a-98f4-c5a88c2785c7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-tjmst\" (UID: \"e6a2716f-443f-4b0a-98f4-c5a88c2785c7\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-tjmst" Jul 06 19:28:56.813359 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:56.813315 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd8kx\" (UniqueName: \"kubernetes.io/projected/e6a2716f-443f-4b0a-98f4-c5a88c2785c7-kube-api-access-nd8kx\") pod \"prometheus-operator-567f7b8b7c-tjmst\" (UID: \"e6a2716f-443f-4b0a-98f4-c5a88c2785c7\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-tjmst" Jul 06 19:28:57.304258 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:57.304158 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e6a2716f-443f-4b0a-98f4-c5a88c2785c7-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-tjmst\" (UID: \"e6a2716f-443f-4b0a-98f4-c5a88c2785c7\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-tjmst" Jul 06 19:28:57.306786 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:57.306757 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e6a2716f-443f-4b0a-98f4-c5a88c2785c7-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-tjmst\" (UID: \"e6a2716f-443f-4b0a-98f4-c5a88c2785c7\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-tjmst" Jul 06 19:28:57.317521 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:57.317489 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gnxgr" event={"ID":"660f827d-c694-49d5-9af0-1458f46f2ffb","Type":"ContainerStarted","Data":"1cfccc6c4542f0e36f8f0391a2142a1257dd49587f19ec7e3ebaf74bdd59ec24"} Jul 06 19:28:57.319318 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:57.319296 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bf496c95-7q6sp" event={"ID":"6b07f9de-449f-4154-b0ee-2d06d6a93313","Type":"ContainerStarted","Data":"83ab831541e3f744ea44f1ec0ab8505e5f7cc1ddae7740fe393266a8064a3f36"} Jul 06 19:28:57.358283 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:57.358240 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gnxgr" podStartSLOduration=33.037721733 podStartE2EDuration="41.358225648s" totalImagesPullingTime="8.320503915s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:28:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:28:57.357808519 +0000 UTC m=+146.177897255" watchObservedRunningTime="2026-07-06 19:28:57.358225648 +0000 UTC m=+146.178314355" Jul 06 19:28:57.388962 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:57.388915 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7bf496c95-7q6sp" podStartSLOduration=1.8275848190000001 podStartE2EDuration="5.388896578s" totalImagesPullingTime="3.561311759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:28:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:28:57.387957919 +0000 UTC m=+146.208046631" watchObservedRunningTime="2026-07-06 19:28:57.388896578 +0000 UTC m=+146.208985291" Jul 06 19:28:57.515149 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:57.515124 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-tjmst" Jul 06 19:28:57.652802 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:57.652772 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-tjmst"] Jul 06 19:28:57.655401 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:28:57.655376 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6a2716f_443f_4b0a_98f4_c5a88c2785c7.slice/crio-11c805875b98a5cdc2e708ccfc4c70cfb9f820e69701bc83bbfa3747f3fb7a26 WatchSource:0}: Error finding container 11c805875b98a5cdc2e708ccfc4c70cfb9f820e69701bc83bbfa3747f3fb7a26: Status 404 returned error can't find the container with id 11c805875b98a5cdc2e708ccfc4c70cfb9f820e69701bc83bbfa3747f3fb7a26 Jul 06 19:28:58.324950 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:58.324876 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-tjmst" event={"ID":"e6a2716f-443f-4b0a-98f4-c5a88c2785c7","Type":"ContainerStarted","Data":"11c805875b98a5cdc2e708ccfc4c70cfb9f820e69701bc83bbfa3747f3fb7a26"} Jul 06 19:28:59.329604 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:59.329563 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-tjmst" event={"ID":"e6a2716f-443f-4b0a-98f4-c5a88c2785c7","Type":"ContainerStarted","Data":"3b4d07fce5205bccdac342b58750669e038e8428c6c958a00c19cd6e68179753"} Jul 06 19:28:59.329994 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:28:59.329644 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-tjmst" event={"ID":"e6a2716f-443f-4b0a-98f4-c5a88c2785c7","Type":"ContainerStarted","Data":"26bab67451c28b7dd7e6fb86bce22e88cbf5f02d5299803aac8c301581a7a057"} Jul 06 19:29:01.224897 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.224838 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-tjmst" podStartSLOduration=4.033486092 podStartE2EDuration="5.224815357s" totalImagesPullingTime="1.191329265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:28:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:28:59.353126082 +0000 UTC m=+148.173214793" watchObservedRunningTime="2026-07-06 19:29:01.224815357 +0000 UTC m=+150.044904067" Jul 06 19:29:01.225374 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.225006 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-dxk9h"] Jul 06 19:29:01.228681 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.228657 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-dxk9h" Jul 06 19:29:01.230809 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.230786 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4s6hk"] Jul 06 19:29:01.231690 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.231241 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-khtwb\"" Jul 06 19:29:01.231690 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.231511 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 06 19:29:01.232096 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.231943 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 06 19:29:01.234261 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.234243 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4s6hk" Jul 06 19:29:01.237557 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.236940 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7b468\"" Jul 06 19:29:01.237557 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.237218 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 19:29:01.237557 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.237396 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 19:29:01.237557 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.237558 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 19:29:01.245939 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.245917 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-dxk9h"] Jul 06 19:29:01.275318 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.275116 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wm4jf"] Jul 06 19:29:01.279867 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.279025 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wm4jf" Jul 06 19:29:01.282212 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.282020 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 06 19:29:01.282212 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.282076 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-4cfgn\"" Jul 06 19:29:01.282212 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.282110 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 06 19:29:01.282681 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.282663 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 06 19:29:01.304644 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.304621 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wm4jf"] Jul 06 19:29:01.335311 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.335289 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phmcg\" (UniqueName: \"kubernetes.io/projected/f2ebcfe7-cf1d-46b8-b531-92f1502af453-kube-api-access-phmcg\") pod \"node-exporter-4s6hk\" (UID: \"f2ebcfe7-cf1d-46b8-b531-92f1502af453\") " pod="openshift-monitoring/node-exporter-4s6hk" Jul 06 19:29:01.335417 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.335325 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f2ebcfe7-cf1d-46b8-b531-92f1502af453-node-exporter-tls\") pod \"node-exporter-4s6hk\" (UID: \"f2ebcfe7-cf1d-46b8-b531-92f1502af453\") " pod="openshift-monitoring/node-exporter-4s6hk" Jul 06 19:29:01.335417 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.335375 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2ebcfe7-cf1d-46b8-b531-92f1502af453-metrics-client-ca\") pod \"node-exporter-4s6hk\" (UID: \"f2ebcfe7-cf1d-46b8-b531-92f1502af453\") " pod="openshift-monitoring/node-exporter-4s6hk" Jul 06 19:29:01.335417 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.335412 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2ebcfe7-cf1d-46b8-b531-92f1502af453-sys\") pod \"node-exporter-4s6hk\" (UID: \"f2ebcfe7-cf1d-46b8-b531-92f1502af453\") " pod="openshift-monitoring/node-exporter-4s6hk" Jul 06 19:29:01.335570 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.335433 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2ebcfe7-cf1d-46b8-b531-92f1502af453-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4s6hk\" (UID: \"f2ebcfe7-cf1d-46b8-b531-92f1502af453\") " pod="openshift-monitoring/node-exporter-4s6hk" Jul 06 19:29:01.335570 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.335457 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f2ebcfe7-cf1d-46b8-b531-92f1502af453-node-exporter-accelerators-collector-config\") pod \"node-exporter-4s6hk\" (UID: \"f2ebcfe7-cf1d-46b8-b531-92f1502af453\") " pod="openshift-monitoring/node-exporter-4s6hk" Jul 06 19:29:01.335570 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.335497 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee3ac022-0831-45e4-95b0-a9e1c2a1b919-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-dxk9h\" (UID: \"ee3ac022-0831-45e4-95b0-a9e1c2a1b919\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-dxk9h" Jul 06 19:29:01.335570 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.335524 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f2ebcfe7-cf1d-46b8-b531-92f1502af453-node-exporter-wtmp\") pod \"node-exporter-4s6hk\" (UID: \"f2ebcfe7-cf1d-46b8-b531-92f1502af453\") " pod="openshift-monitoring/node-exporter-4s6hk" Jul 06 19:29:01.335758 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.335586 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f2ebcfe7-cf1d-46b8-b531-92f1502af453-root\") pod \"node-exporter-4s6hk\" (UID: \"f2ebcfe7-cf1d-46b8-b531-92f1502af453\") " pod="openshift-monitoring/node-exporter-4s6hk" Jul 06 19:29:01.335758 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.335625 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqhzb\" (UniqueName: \"kubernetes.io/projected/ee3ac022-0831-45e4-95b0-a9e1c2a1b919-kube-api-access-gqhzb\") pod \"openshift-state-metrics-54cf96dfcf-dxk9h\" (UID: \"ee3ac022-0831-45e4-95b0-a9e1c2a1b919\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-dxk9h" Jul 06 19:29:01.335758 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.335651 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee3ac022-0831-45e4-95b0-a9e1c2a1b919-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-dxk9h\" (UID: \"ee3ac022-0831-45e4-95b0-a9e1c2a1b919\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-dxk9h" Jul 06 19:29:01.335758 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.335687 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ee3ac022-0831-45e4-95b0-a9e1c2a1b919-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-dxk9h\" (UID: \"ee3ac022-0831-45e4-95b0-a9e1c2a1b919\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-dxk9h" Jul 06 19:29:01.335758 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.335713 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f2ebcfe7-cf1d-46b8-b531-92f1502af453-node-exporter-textfile\") pod \"node-exporter-4s6hk\" (UID: \"f2ebcfe7-cf1d-46b8-b531-92f1502af453\") " pod="openshift-monitoring/node-exporter-4s6hk" Jul 06 19:29:01.438159 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.436381 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f2ebcfe7-cf1d-46b8-b531-92f1502af453-node-exporter-wtmp\") pod \"node-exporter-4s6hk\" (UID: \"f2ebcfe7-cf1d-46b8-b531-92f1502af453\") " pod="openshift-monitoring/node-exporter-4s6hk" Jul 06 19:29:01.438159 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.436433 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f2ebcfe7-cf1d-46b8-b531-92f1502af453-root\") pod \"node-exporter-4s6hk\" (UID: \"f2ebcfe7-cf1d-46b8-b531-92f1502af453\") " pod="openshift-monitoring/node-exporter-4s6hk" Jul 06 19:29:01.438159 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.436466 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqhzb\" (UniqueName: \"kubernetes.io/projected/ee3ac022-0831-45e4-95b0-a9e1c2a1b919-kube-api-access-gqhzb\") pod \"openshift-state-metrics-54cf96dfcf-dxk9h\" (UID: \"ee3ac022-0831-45e4-95b0-a9e1c2a1b919\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-dxk9h" Jul 06 19:29:01.438159 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.436493 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee3ac022-0831-45e4-95b0-a9e1c2a1b919-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-dxk9h\" (UID: \"ee3ac022-0831-45e4-95b0-a9e1c2a1b919\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-dxk9h" Jul 06 19:29:01.438159 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.436519 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ee3ac022-0831-45e4-95b0-a9e1c2a1b919-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-dxk9h\" (UID: \"ee3ac022-0831-45e4-95b0-a9e1c2a1b919\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-dxk9h" Jul 06 19:29:01.438159 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.436544 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f2ebcfe7-cf1d-46b8-b531-92f1502af453-node-exporter-textfile\") pod \"node-exporter-4s6hk\" (UID: \"f2ebcfe7-cf1d-46b8-b531-92f1502af453\") " pod="openshift-monitoring/node-exporter-4s6hk" Jul 06 19:29:01.438159 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.436575 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phmcg\" (UniqueName: \"kubernetes.io/projected/f2ebcfe7-cf1d-46b8-b531-92f1502af453-kube-api-access-phmcg\") pod \"node-exporter-4s6hk\" (UID: \"f2ebcfe7-cf1d-46b8-b531-92f1502af453\") " pod="openshift-monitoring/node-exporter-4s6hk" Jul 06 19:29:01.438159 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.436605 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/12d78a18-d2c1-429d-81bb-14f3b8e4c21e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-wm4jf\" (UID: \"12d78a18-d2c1-429d-81bb-14f3b8e4c21e\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wm4jf" Jul 06 19:29:01.438159 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.436631 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/12d78a18-d2c1-429d-81bb-14f3b8e4c21e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-wm4jf\" (UID: \"12d78a18-d2c1-429d-81bb-14f3b8e4c21e\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wm4jf" Jul 06 19:29:01.438159 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.436657 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f2ebcfe7-cf1d-46b8-b531-92f1502af453-node-exporter-tls\") pod \"node-exporter-4s6hk\" (UID: \"f2ebcfe7-cf1d-46b8-b531-92f1502af453\") " pod="openshift-monitoring/node-exporter-4s6hk" Jul 06 19:29:01.438159 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.436707 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2ebcfe7-cf1d-46b8-b531-92f1502af453-metrics-client-ca\") pod \"node-exporter-4s6hk\" (UID: \"f2ebcfe7-cf1d-46b8-b531-92f1502af453\") " pod="openshift-monitoring/node-exporter-4s6hk" Jul 06 19:29:01.438159 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.436738 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/12d78a18-d2c1-429d-81bb-14f3b8e4c21e-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-wm4jf\" (UID: \"12d78a18-d2c1-429d-81bb-14f3b8e4c21e\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wm4jf" Jul 06 19:29:01.438159 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.436762 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/12d78a18-d2c1-429d-81bb-14f3b8e4c21e-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-wm4jf\" (UID: \"12d78a18-d2c1-429d-81bb-14f3b8e4c21e\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wm4jf" Jul 06 19:29:01.438907 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.436792 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2ebcfe7-cf1d-46b8-b531-92f1502af453-sys\") pod \"node-exporter-4s6hk\" (UID: \"f2ebcfe7-cf1d-46b8-b531-92f1502af453\") " pod="openshift-monitoring/node-exporter-4s6hk" Jul 06 19:29:01.438907 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.436815 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2ebcfe7-cf1d-46b8-b531-92f1502af453-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4s6hk\" (UID: \"f2ebcfe7-cf1d-46b8-b531-92f1502af453\") " pod="openshift-monitoring/node-exporter-4s6hk" Jul 06 19:29:01.438907 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.436841 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f2ebcfe7-cf1d-46b8-b531-92f1502af453-node-exporter-accelerators-collector-config\") pod \"node-exporter-4s6hk\" (UID: \"f2ebcfe7-cf1d-46b8-b531-92f1502af453\") " pod="openshift-monitoring/node-exporter-4s6hk" Jul 06 19:29:01.438907 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.436864 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12d78a18-d2c1-429d-81bb-14f3b8e4c21e-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-wm4jf\" (UID: \"12d78a18-d2c1-429d-81bb-14f3b8e4c21e\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wm4jf" Jul 06 19:29:01.438907 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.436893 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t59k\" (UniqueName: \"kubernetes.io/projected/12d78a18-d2c1-429d-81bb-14f3b8e4c21e-kube-api-access-6t59k\") pod \"kube-state-metrics-7c9dbb8b7d-wm4jf\" (UID: \"12d78a18-d2c1-429d-81bb-14f3b8e4c21e\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wm4jf" Jul 06 19:29:01.438907 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.436936 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee3ac022-0831-45e4-95b0-a9e1c2a1b919-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-dxk9h\" (UID: \"ee3ac022-0831-45e4-95b0-a9e1c2a1b919\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-dxk9h" Jul 06 19:29:01.438907 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.437335 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f2ebcfe7-cf1d-46b8-b531-92f1502af453-node-exporter-wtmp\") pod \"node-exporter-4s6hk\" (UID: \"f2ebcfe7-cf1d-46b8-b531-92f1502af453\") " pod="openshift-monitoring/node-exporter-4s6hk" Jul 06 19:29:01.438907 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.437384 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f2ebcfe7-cf1d-46b8-b531-92f1502af453-root\") pod \"node-exporter-4s6hk\" (UID: \"f2ebcfe7-cf1d-46b8-b531-92f1502af453\") " pod="openshift-monitoring/node-exporter-4s6hk" Jul 06 19:29:01.438907 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:29:01.437667 2575 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 06 19:29:01.438907 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:29:01.437718 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee3ac022-0831-45e4-95b0-a9e1c2a1b919-openshift-state-metrics-tls podName:ee3ac022-0831-45e4-95b0-a9e1c2a1b919 nodeName:}" failed. No retries permitted until 2026-07-06 19:29:01.937700948 +0000 UTC m=+150.757789644 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/ee3ac022-0831-45e4-95b0-a9e1c2a1b919-openshift-state-metrics-tls") pod "openshift-state-metrics-54cf96dfcf-dxk9h" (UID: "ee3ac022-0831-45e4-95b0-a9e1c2a1b919") : secret "openshift-state-metrics-tls" not found Jul 06 19:29:01.438907 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.437873 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2ebcfe7-cf1d-46b8-b531-92f1502af453-metrics-client-ca\") pod \"node-exporter-4s6hk\" (UID: \"f2ebcfe7-cf1d-46b8-b531-92f1502af453\") " pod="openshift-monitoring/node-exporter-4s6hk" Jul 06 19:29:01.438907 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:29:01.437976 2575 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 06 19:29:01.438907 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:29:01.438021 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2ebcfe7-cf1d-46b8-b531-92f1502af453-node-exporter-tls podName:f2ebcfe7-cf1d-46b8-b531-92f1502af453 nodeName:}" failed. No retries permitted until 2026-07-06 19:29:01.938005529 +0000 UTC m=+150.758094223 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/f2ebcfe7-cf1d-46b8-b531-92f1502af453-node-exporter-tls") pod "node-exporter-4s6hk" (UID: "f2ebcfe7-cf1d-46b8-b531-92f1502af453") : secret "node-exporter-tls" not found Jul 06 19:29:01.439615 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.438085 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2ebcfe7-cf1d-46b8-b531-92f1502af453-sys\") pod \"node-exporter-4s6hk\" (UID: \"f2ebcfe7-cf1d-46b8-b531-92f1502af453\") " pod="openshift-monitoring/node-exporter-4s6hk" Jul 06 19:29:01.439615 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.439084 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f2ebcfe7-cf1d-46b8-b531-92f1502af453-node-exporter-accelerators-collector-config\") pod \"node-exporter-4s6hk\" (UID: \"f2ebcfe7-cf1d-46b8-b531-92f1502af453\") " pod="openshift-monitoring/node-exporter-4s6hk" Jul 06 19:29:01.439615 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.439243 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f2ebcfe7-cf1d-46b8-b531-92f1502af453-node-exporter-textfile\") pod \"node-exporter-4s6hk\" (UID: \"f2ebcfe7-cf1d-46b8-b531-92f1502af453\") " pod="openshift-monitoring/node-exporter-4s6hk" Jul 06 19:29:01.440350 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.440298 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee3ac022-0831-45e4-95b0-a9e1c2a1b919-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-dxk9h\" (UID: \"ee3ac022-0831-45e4-95b0-a9e1c2a1b919\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-dxk9h" Jul 06 19:29:01.440649 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.440611 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2ebcfe7-cf1d-46b8-b531-92f1502af453-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4s6hk\" (UID: \"f2ebcfe7-cf1d-46b8-b531-92f1502af453\") " pod="openshift-monitoring/node-exporter-4s6hk" Jul 06 19:29:01.442829 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.442805 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ee3ac022-0831-45e4-95b0-a9e1c2a1b919-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-dxk9h\" (UID: \"ee3ac022-0831-45e4-95b0-a9e1c2a1b919\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-dxk9h" Jul 06 19:29:01.456917 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.456893 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phmcg\" (UniqueName: \"kubernetes.io/projected/f2ebcfe7-cf1d-46b8-b531-92f1502af453-kube-api-access-phmcg\") pod \"node-exporter-4s6hk\" (UID: \"f2ebcfe7-cf1d-46b8-b531-92f1502af453\") " pod="openshift-monitoring/node-exporter-4s6hk" Jul 06 19:29:01.457448 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.457428 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqhzb\" (UniqueName: \"kubernetes.io/projected/ee3ac022-0831-45e4-95b0-a9e1c2a1b919-kube-api-access-gqhzb\") pod \"openshift-state-metrics-54cf96dfcf-dxk9h\" (UID: \"ee3ac022-0831-45e4-95b0-a9e1c2a1b919\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-dxk9h" Jul 06 19:29:01.537689 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.537632 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/12d78a18-d2c1-429d-81bb-14f3b8e4c21e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-wm4jf\" (UID: \"12d78a18-d2c1-429d-81bb-14f3b8e4c21e\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wm4jf" Jul 06 19:29:01.537689 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.537667 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/12d78a18-d2c1-429d-81bb-14f3b8e4c21e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-wm4jf\" (UID: \"12d78a18-d2c1-429d-81bb-14f3b8e4c21e\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wm4jf" Jul 06 19:29:01.537843 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.537732 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/12d78a18-d2c1-429d-81bb-14f3b8e4c21e-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-wm4jf\" (UID: \"12d78a18-d2c1-429d-81bb-14f3b8e4c21e\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wm4jf" Jul 06 19:29:01.537843 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.537758 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/12d78a18-d2c1-429d-81bb-14f3b8e4c21e-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-wm4jf\" (UID: \"12d78a18-d2c1-429d-81bb-14f3b8e4c21e\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wm4jf" Jul 06 19:29:01.537843 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.537791 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12d78a18-d2c1-429d-81bb-14f3b8e4c21e-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-wm4jf\" (UID: \"12d78a18-d2c1-429d-81bb-14f3b8e4c21e\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wm4jf" Jul 06 19:29:01.537843 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.537820 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t59k\" (UniqueName: \"kubernetes.io/projected/12d78a18-d2c1-429d-81bb-14f3b8e4c21e-kube-api-access-6t59k\") pod \"kube-state-metrics-7c9dbb8b7d-wm4jf\" (UID: \"12d78a18-d2c1-429d-81bb-14f3b8e4c21e\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wm4jf" Jul 06 19:29:01.538809 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:29:01.538584 2575 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 06 19:29:01.538809 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:29:01.538646 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12d78a18-d2c1-429d-81bb-14f3b8e4c21e-kube-state-metrics-tls podName:12d78a18-d2c1-429d-81bb-14f3b8e4c21e nodeName:}" failed. No retries permitted until 2026-07-06 19:29:02.038627916 +0000 UTC m=+150.858716612 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/12d78a18-d2c1-429d-81bb-14f3b8e4c21e-kube-state-metrics-tls") pod "kube-state-metrics-7c9dbb8b7d-wm4jf" (UID: "12d78a18-d2c1-429d-81bb-14f3b8e4c21e") : secret "kube-state-metrics-tls" not found Jul 06 19:29:01.538956 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.538831 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/12d78a18-d2c1-429d-81bb-14f3b8e4c21e-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-wm4jf\" (UID: \"12d78a18-d2c1-429d-81bb-14f3b8e4c21e\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wm4jf" Jul 06 19:29:01.539302 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.539250 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12d78a18-d2c1-429d-81bb-14f3b8e4c21e-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-wm4jf\" (UID: \"12d78a18-d2c1-429d-81bb-14f3b8e4c21e\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wm4jf" Jul 06 19:29:01.539302 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.539264 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/12d78a18-d2c1-429d-81bb-14f3b8e4c21e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-wm4jf\" (UID: \"12d78a18-d2c1-429d-81bb-14f3b8e4c21e\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wm4jf" Jul 06 19:29:01.541015 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.540991 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/12d78a18-d2c1-429d-81bb-14f3b8e4c21e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-wm4jf\" (UID: \"12d78a18-d2c1-429d-81bb-14f3b8e4c21e\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wm4jf" Jul 06 19:29:01.565777 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.565721 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t59k\" (UniqueName: \"kubernetes.io/projected/12d78a18-d2c1-429d-81bb-14f3b8e4c21e-kube-api-access-6t59k\") pod \"kube-state-metrics-7c9dbb8b7d-wm4jf\" (UID: \"12d78a18-d2c1-429d-81bb-14f3b8e4c21e\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wm4jf" Jul 06 19:29:01.941574 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.941544 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee3ac022-0831-45e4-95b0-a9e1c2a1b919-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-dxk9h\" (UID: \"ee3ac022-0831-45e4-95b0-a9e1c2a1b919\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-dxk9h" Jul 06 19:29:01.941750 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.941592 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f2ebcfe7-cf1d-46b8-b531-92f1502af453-node-exporter-tls\") pod \"node-exporter-4s6hk\" (UID: \"f2ebcfe7-cf1d-46b8-b531-92f1502af453\") " pod="openshift-monitoring/node-exporter-4s6hk" Jul 06 19:29:01.941810 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:29:01.941749 2575 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 06 19:29:01.941864 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:29:01.941813 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee3ac022-0831-45e4-95b0-a9e1c2a1b919-openshift-state-metrics-tls podName:ee3ac022-0831-45e4-95b0-a9e1c2a1b919 nodeName:}" failed. No retries permitted until 2026-07-06 19:29:02.941793823 +0000 UTC m=+151.761882528 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/ee3ac022-0831-45e4-95b0-a9e1c2a1b919-openshift-state-metrics-tls") pod "openshift-state-metrics-54cf96dfcf-dxk9h" (UID: "ee3ac022-0831-45e4-95b0-a9e1c2a1b919") : secret "openshift-state-metrics-tls" not found Jul 06 19:29:01.944469 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:01.944444 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f2ebcfe7-cf1d-46b8-b531-92f1502af453-node-exporter-tls\") pod \"node-exporter-4s6hk\" (UID: \"f2ebcfe7-cf1d-46b8-b531-92f1502af453\") " pod="openshift-monitoring/node-exporter-4s6hk" Jul 06 19:29:02.042866 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:02.042786 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/12d78a18-d2c1-429d-81bb-14f3b8e4c21e-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-wm4jf\" (UID: \"12d78a18-d2c1-429d-81bb-14f3b8e4c21e\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wm4jf" Jul 06 19:29:02.045659 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:02.045636 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/12d78a18-d2c1-429d-81bb-14f3b8e4c21e-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-wm4jf\" (UID: \"12d78a18-d2c1-429d-81bb-14f3b8e4c21e\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wm4jf" Jul 06 19:29:02.150425 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:02.150395 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4s6hk" Jul 06 19:29:02.162323 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:29:02.162304 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2ebcfe7_cf1d_46b8_b531_92f1502af453.slice/crio-2664805fc6e8bbd050fc36c90d6f72e5e5f4005852b41f53e7b56bd25e202fe3 WatchSource:0}: Error finding container 2664805fc6e8bbd050fc36c90d6f72e5e5f4005852b41f53e7b56bd25e202fe3: Status 404 returned error can't find the container with id 2664805fc6e8bbd050fc36c90d6f72e5e5f4005852b41f53e7b56bd25e202fe3 Jul 06 19:29:02.191110 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:02.191086 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wm4jf" Jul 06 19:29:02.343139 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:02.343099 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4s6hk" event={"ID":"f2ebcfe7-cf1d-46b8-b531-92f1502af453","Type":"ContainerStarted","Data":"2664805fc6e8bbd050fc36c90d6f72e5e5f4005852b41f53e7b56bd25e202fe3"} Jul 06 19:29:02.358295 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:02.358163 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wm4jf"] Jul 06 19:29:02.360479 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:29:02.360442 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12d78a18_d2c1_429d_81bb_14f3b8e4c21e.slice/crio-c3d027886d68a0f893251dfd1adacf607deac466bd2a1ae95deaab56ecb813cb WatchSource:0}: Error finding container c3d027886d68a0f893251dfd1adacf607deac466bd2a1ae95deaab56ecb813cb: Status 404 returned error can't find the container with id c3d027886d68a0f893251dfd1adacf607deac466bd2a1ae95deaab56ecb813cb Jul 06 19:29:02.950026 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:02.949985 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee3ac022-0831-45e4-95b0-a9e1c2a1b919-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-dxk9h\" (UID: \"ee3ac022-0831-45e4-95b0-a9e1c2a1b919\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-dxk9h" Jul 06 19:29:02.952983 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:02.952959 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee3ac022-0831-45e4-95b0-a9e1c2a1b919-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-dxk9h\" (UID: \"ee3ac022-0831-45e4-95b0-a9e1c2a1b919\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-dxk9h" Jul 06 19:29:03.042081 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:03.042052 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-dxk9h" Jul 06 19:29:03.164902 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:03.164350 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7bf496c95-7q6sp" Jul 06 19:29:03.164902 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:03.164822 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bf496c95-7q6sp" Jul 06 19:29:03.170805 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:03.170542 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7bf496c95-7q6sp" Jul 06 19:29:03.272087 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:03.272009 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-dxk9h"] Jul 06 19:29:03.275752 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:29:03.275724 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee3ac022_0831_45e4_95b0_a9e1c2a1b919.slice/crio-6f89a1989d47dc7c93f5c57d4c22d1480304e707bc8c7afaf8b01ec76cb0421c WatchSource:0}: Error finding container 6f89a1989d47dc7c93f5c57d4c22d1480304e707bc8c7afaf8b01ec76cb0421c: Status 404 returned error can't find the container with id 6f89a1989d47dc7c93f5c57d4c22d1480304e707bc8c7afaf8b01ec76cb0421c Jul 06 19:29:03.349747 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:03.349717 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-dxk9h" event={"ID":"ee3ac022-0831-45e4-95b0-a9e1c2a1b919","Type":"ContainerStarted","Data":"6f89a1989d47dc7c93f5c57d4c22d1480304e707bc8c7afaf8b01ec76cb0421c"} Jul 06 19:29:03.351799 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:03.351571 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4s6hk" event={"ID":"f2ebcfe7-cf1d-46b8-b531-92f1502af453","Type":"ContainerStarted","Data":"705bd1178871ef3143597ed2a33ef46a7ea29d3701b6dad99d1a9b4cdb4f9bfb"} Jul 06 19:29:03.352900 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:03.352871 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wm4jf" event={"ID":"12d78a18-d2c1-429d-81bb-14f3b8e4c21e","Type":"ContainerStarted","Data":"c3d027886d68a0f893251dfd1adacf607deac466bd2a1ae95deaab56ecb813cb"} Jul 06 19:29:03.358736 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:03.358688 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7bf496c95-7q6sp" Jul 06 19:29:04.189618 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.189581 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7c7cf86bbd-ck249"] Jul 06 19:29:04.193888 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.193867 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7c7cf86bbd-ck249" Jul 06 19:29:04.198764 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.198713 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 06 19:29:04.198893 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.198864 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 06 19:29:04.199523 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.199303 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 06 19:29:04.199523 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.199513 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-tn4pg\"" Jul 06 19:29:04.200125 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.199922 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 06 19:29:04.200125 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.199928 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 06 19:29:04.200351 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.200332 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-1onosg36m1ajj\"" Jul 06 19:29:04.218931 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.218860 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7c7cf86bbd-ck249"] Jul 06 19:29:04.261012 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.260987 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cf8e44b4-157b-43f6-b854-5b39e1e718c9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c7cf86bbd-ck249\" (UID: \"cf8e44b4-157b-43f6-b854-5b39e1e718c9\") " pod="openshift-monitoring/thanos-querier-7c7cf86bbd-ck249" Jul 06 19:29:04.261153 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.261056 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmmhg\" (UniqueName: \"kubernetes.io/projected/cf8e44b4-157b-43f6-b854-5b39e1e718c9-kube-api-access-dmmhg\") pod \"thanos-querier-7c7cf86bbd-ck249\" (UID: \"cf8e44b4-157b-43f6-b854-5b39e1e718c9\") " pod="openshift-monitoring/thanos-querier-7c7cf86bbd-ck249" Jul 06 19:29:04.261153 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.261108 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cf8e44b4-157b-43f6-b854-5b39e1e718c9-secret-grpc-tls\") pod \"thanos-querier-7c7cf86bbd-ck249\" (UID: \"cf8e44b4-157b-43f6-b854-5b39e1e718c9\") " pod="openshift-monitoring/thanos-querier-7c7cf86bbd-ck249" Jul 06 19:29:04.261153 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.261134 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cf8e44b4-157b-43f6-b854-5b39e1e718c9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c7cf86bbd-ck249\" (UID: \"cf8e44b4-157b-43f6-b854-5b39e1e718c9\") " pod="openshift-monitoring/thanos-querier-7c7cf86bbd-ck249" Jul 06 19:29:04.261480 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.261204 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf8e44b4-157b-43f6-b854-5b39e1e718c9-metrics-client-ca\") pod \"thanos-querier-7c7cf86bbd-ck249\" (UID: \"cf8e44b4-157b-43f6-b854-5b39e1e718c9\") " pod="openshift-monitoring/thanos-querier-7c7cf86bbd-ck249" Jul 06 19:29:04.261480 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.261256 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cf8e44b4-157b-43f6-b854-5b39e1e718c9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c7cf86bbd-ck249\" (UID: \"cf8e44b4-157b-43f6-b854-5b39e1e718c9\") " pod="openshift-monitoring/thanos-querier-7c7cf86bbd-ck249" Jul 06 19:29:04.261480 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.261295 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cf8e44b4-157b-43f6-b854-5b39e1e718c9-secret-thanos-querier-tls\") pod \"thanos-querier-7c7cf86bbd-ck249\" (UID: \"cf8e44b4-157b-43f6-b854-5b39e1e718c9\") " pod="openshift-monitoring/thanos-querier-7c7cf86bbd-ck249" Jul 06 19:29:04.261480 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.261316 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cf8e44b4-157b-43f6-b854-5b39e1e718c9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c7cf86bbd-ck249\" (UID: \"cf8e44b4-157b-43f6-b854-5b39e1e718c9\") " pod="openshift-monitoring/thanos-querier-7c7cf86bbd-ck249" Jul 06 19:29:04.363349 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.362487 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cf8e44b4-157b-43f6-b854-5b39e1e718c9-secret-thanos-querier-tls\") pod \"thanos-querier-7c7cf86bbd-ck249\" (UID: \"cf8e44b4-157b-43f6-b854-5b39e1e718c9\") " pod="openshift-monitoring/thanos-querier-7c7cf86bbd-ck249" Jul 06 19:29:04.363349 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.362571 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cf8e44b4-157b-43f6-b854-5b39e1e718c9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c7cf86bbd-ck249\" (UID: \"cf8e44b4-157b-43f6-b854-5b39e1e718c9\") " pod="openshift-monitoring/thanos-querier-7c7cf86bbd-ck249" Jul 06 19:29:04.363349 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.362615 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cf8e44b4-157b-43f6-b854-5b39e1e718c9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c7cf86bbd-ck249\" (UID: \"cf8e44b4-157b-43f6-b854-5b39e1e718c9\") " pod="openshift-monitoring/thanos-querier-7c7cf86bbd-ck249" Jul 06 19:29:04.363349 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.362683 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmmhg\" (UniqueName: \"kubernetes.io/projected/cf8e44b4-157b-43f6-b854-5b39e1e718c9-kube-api-access-dmmhg\") pod \"thanos-querier-7c7cf86bbd-ck249\" (UID: \"cf8e44b4-157b-43f6-b854-5b39e1e718c9\") " pod="openshift-monitoring/thanos-querier-7c7cf86bbd-ck249" Jul 06 19:29:04.363349 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.362742 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cf8e44b4-157b-43f6-b854-5b39e1e718c9-secret-grpc-tls\") pod \"thanos-querier-7c7cf86bbd-ck249\" (UID: \"cf8e44b4-157b-43f6-b854-5b39e1e718c9\") " pod="openshift-monitoring/thanos-querier-7c7cf86bbd-ck249" Jul 06 19:29:04.363349 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.362777 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cf8e44b4-157b-43f6-b854-5b39e1e718c9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c7cf86bbd-ck249\" (UID: \"cf8e44b4-157b-43f6-b854-5b39e1e718c9\") " pod="openshift-monitoring/thanos-querier-7c7cf86bbd-ck249" Jul 06 19:29:04.363349 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.362834 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf8e44b4-157b-43f6-b854-5b39e1e718c9-metrics-client-ca\") pod \"thanos-querier-7c7cf86bbd-ck249\" (UID: \"cf8e44b4-157b-43f6-b854-5b39e1e718c9\") " pod="openshift-monitoring/thanos-querier-7c7cf86bbd-ck249" Jul 06 19:29:04.363349 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.362870 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cf8e44b4-157b-43f6-b854-5b39e1e718c9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c7cf86bbd-ck249\" (UID: \"cf8e44b4-157b-43f6-b854-5b39e1e718c9\") " pod="openshift-monitoring/thanos-querier-7c7cf86bbd-ck249" Jul 06 19:29:04.365475 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.365450 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf8e44b4-157b-43f6-b854-5b39e1e718c9-metrics-client-ca\") pod \"thanos-querier-7c7cf86bbd-ck249\" (UID: \"cf8e44b4-157b-43f6-b854-5b39e1e718c9\") " pod="openshift-monitoring/thanos-querier-7c7cf86bbd-ck249" Jul 06 19:29:04.369293 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.369267 2575 generic.go:356] "Generic (PLEG): container finished" podID="f2ebcfe7-cf1d-46b8-b531-92f1502af453" containerID="705bd1178871ef3143597ed2a33ef46a7ea29d3701b6dad99d1a9b4cdb4f9bfb" exitCode=0 Jul 06 19:29:04.369538 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.369519 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4s6hk" event={"ID":"f2ebcfe7-cf1d-46b8-b531-92f1502af453","Type":"ContainerDied","Data":"705bd1178871ef3143597ed2a33ef46a7ea29d3701b6dad99d1a9b4cdb4f9bfb"} Jul 06 19:29:04.369705 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.369691 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4s6hk" event={"ID":"f2ebcfe7-cf1d-46b8-b531-92f1502af453","Type":"ContainerStarted","Data":"113c5214adf0335acf2cc131a3defdf338df9b88eea80803d2210ae0ad7bb2f3"} Jul 06 19:29:04.369816 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.369804 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4s6hk" event={"ID":"f2ebcfe7-cf1d-46b8-b531-92f1502af453","Type":"ContainerStarted","Data":"053fac9c45e512c69c89c991c679ba8ae07a54e8f01a3f0e7e5ca1faec74e712"} Jul 06 19:29:04.371282 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.371257 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cf8e44b4-157b-43f6-b854-5b39e1e718c9-secret-grpc-tls\") pod \"thanos-querier-7c7cf86bbd-ck249\" (UID: \"cf8e44b4-157b-43f6-b854-5b39e1e718c9\") " pod="openshift-monitoring/thanos-querier-7c7cf86bbd-ck249" Jul 06 19:29:04.373602 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.373575 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cf8e44b4-157b-43f6-b854-5b39e1e718c9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c7cf86bbd-ck249\" (UID: \"cf8e44b4-157b-43f6-b854-5b39e1e718c9\") " pod="openshift-monitoring/thanos-querier-7c7cf86bbd-ck249" Jul 06 19:29:04.374334 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.374310 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cf8e44b4-157b-43f6-b854-5b39e1e718c9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c7cf86bbd-ck249\" (UID: \"cf8e44b4-157b-43f6-b854-5b39e1e718c9\") " pod="openshift-monitoring/thanos-querier-7c7cf86bbd-ck249" Jul 06 19:29:04.375911 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.374890 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cf8e44b4-157b-43f6-b854-5b39e1e718c9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c7cf86bbd-ck249\" (UID: \"cf8e44b4-157b-43f6-b854-5b39e1e718c9\") " pod="openshift-monitoring/thanos-querier-7c7cf86bbd-ck249" Jul 06 19:29:04.375911 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.375877 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cf8e44b4-157b-43f6-b854-5b39e1e718c9-secret-thanos-querier-tls\") pod \"thanos-querier-7c7cf86bbd-ck249\" (UID: \"cf8e44b4-157b-43f6-b854-5b39e1e718c9\") " pod="openshift-monitoring/thanos-querier-7c7cf86bbd-ck249" Jul 06 19:29:04.376824 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.376803 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmmhg\" (UniqueName: \"kubernetes.io/projected/cf8e44b4-157b-43f6-b854-5b39e1e718c9-kube-api-access-dmmhg\") pod \"thanos-querier-7c7cf86bbd-ck249\" (UID: \"cf8e44b4-157b-43f6-b854-5b39e1e718c9\") " pod="openshift-monitoring/thanos-querier-7c7cf86bbd-ck249" Jul 06 19:29:04.378390 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.378278 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wm4jf" event={"ID":"12d78a18-d2c1-429d-81bb-14f3b8e4c21e","Type":"ContainerStarted","Data":"2afc40510553fdd848c4f9ded785f0a8ec67ee5fedf77dfb59d1aa06d05151d5"} Jul 06 19:29:04.378390 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.378329 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wm4jf" event={"ID":"12d78a18-d2c1-429d-81bb-14f3b8e4c21e","Type":"ContainerStarted","Data":"39760c48cd88fb9f8d3b1fe5278b095a5b65f0dd4dd88c769311dcc56c1751a6"} Jul 06 19:29:04.378390 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.378359 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wm4jf" event={"ID":"12d78a18-d2c1-429d-81bb-14f3b8e4c21e","Type":"ContainerStarted","Data":"ab4bbe0f9a192cc4b44dbf49589915d030e73db2c50cd093b4eaf324e543a149"} Jul 06 19:29:04.379507 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.379483 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cf8e44b4-157b-43f6-b854-5b39e1e718c9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c7cf86bbd-ck249\" (UID: \"cf8e44b4-157b-43f6-b854-5b39e1e718c9\") " pod="openshift-monitoring/thanos-querier-7c7cf86bbd-ck249" Jul 06 19:29:04.381373 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.381307 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-dxk9h" event={"ID":"ee3ac022-0831-45e4-95b0-a9e1c2a1b919","Type":"ContainerStarted","Data":"47dff750b4799484132759bd1056db88159d020a4667569d9463dcdd17759be5"} Jul 06 19:29:04.381373 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.381358 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-dxk9h" event={"ID":"ee3ac022-0831-45e4-95b0-a9e1c2a1b919","Type":"ContainerStarted","Data":"17a0d7b27244fc59bf71524f3117fc2ed5130da0506c36cf431e5a70b38a8e05"} Jul 06 19:29:04.394677 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.394625 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4s6hk" podStartSLOduration=2.314081427 podStartE2EDuration="3.394608891s" totalImagesPullingTime="997.353237ms" totalInitContainerRuntime="83.174227ms" isStatefulPod=true podCreationTimestamp="2026-07-06 19:29:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:29:04.394125129 +0000 UTC m=+153.214213877" watchObservedRunningTime="2026-07-06 19:29:04.394608891 +0000 UTC m=+153.214697602" Jul 06 19:29:04.419292 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.419248 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wm4jf" podStartSLOduration=2.026016083 podStartE2EDuration="3.419231583s" totalImagesPullingTime="1.3932155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:29:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:29:04.418705032 +0000 UTC m=+153.238793744" watchObservedRunningTime="2026-07-06 19:29:04.419231583 +0000 UTC m=+153.239320294" Jul 06 19:29:04.505796 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.505531 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7c7cf86bbd-ck249" Jul 06 19:29:04.677539 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:04.677480 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7c7cf86bbd-ck249"] Jul 06 19:29:04.860881 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:29:04.860850 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf8e44b4_157b_43f6_b854_5b39e1e718c9.slice/crio-bd80ed57368087fe213c19eeb6216bda9b7007891dd84aa60709bd888de9821f WatchSource:0}: Error finding container bd80ed57368087fe213c19eeb6216bda9b7007891dd84aa60709bd888de9821f: Status 404 returned error can't find the container with id bd80ed57368087fe213c19eeb6216bda9b7007891dd84aa60709bd888de9821f Jul 06 19:29:05.387023 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:05.386984 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-dxk9h" event={"ID":"ee3ac022-0831-45e4-95b0-a9e1c2a1b919","Type":"ContainerStarted","Data":"76536475def0169d873a972ee0ac6d4d8e1235dd6415c4a6f90214eda450c184"} Jul 06 19:29:05.388232 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:05.388207 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c7cf86bbd-ck249" event={"ID":"cf8e44b4-157b-43f6-b854-5b39e1e718c9","Type":"ContainerStarted","Data":"bd80ed57368087fe213c19eeb6216bda9b7007891dd84aa60709bd888de9821f"} Jul 06 19:29:05.411134 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:05.411092 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-dxk9h" podStartSLOduration=3.285105354 podStartE2EDuration="4.411075934s" totalImagesPullingTime="1.12597058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:29:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:29:05.410025278 +0000 UTC m=+154.230114001" watchObservedRunningTime="2026-07-06 19:29:05.411075934 +0000 UTC m=+154.231164646" Jul 06 19:29:05.752014 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:05.751932 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5f5769b66b-86k92"] Jul 06 19:29:05.755725 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:05.755587 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5f5769b66b-86k92" Jul 06 19:29:05.758855 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:05.758453 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 06 19:29:05.758855 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:05.758580 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-cc7aqoop9qfeq\"" Jul 06 19:29:05.758855 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:05.758698 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 06 19:29:05.758855 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:05.758808 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 06 19:29:05.759568 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:05.759549 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 06 19:29:05.759663 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:05.759630 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-tklrk\"" Jul 06 19:29:05.769939 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:05.769589 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5f5769b66b-86k92"] Jul 06 19:29:05.774251 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:05.774216 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2zlf\" (UniqueName: \"kubernetes.io/projected/9c0394d2-076a-4bef-9198-be9301946e20-kube-api-access-q2zlf\") pod \"metrics-server-5f5769b66b-86k92\" (UID: \"9c0394d2-076a-4bef-9198-be9301946e20\") " pod="openshift-monitoring/metrics-server-5f5769b66b-86k92" Jul 06 19:29:05.774339 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:05.774258 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c0394d2-076a-4bef-9198-be9301946e20-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f5769b66b-86k92\" (UID: \"9c0394d2-076a-4bef-9198-be9301946e20\") " pod="openshift-monitoring/metrics-server-5f5769b66b-86k92" Jul 06 19:29:05.774412 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:05.774352 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9c0394d2-076a-4bef-9198-be9301946e20-metrics-server-audit-profiles\") pod \"metrics-server-5f5769b66b-86k92\" (UID: \"9c0394d2-076a-4bef-9198-be9301946e20\") " pod="openshift-monitoring/metrics-server-5f5769b66b-86k92" Jul 06 19:29:05.774412 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:05.774388 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9c0394d2-076a-4bef-9198-be9301946e20-audit-log\") pod \"metrics-server-5f5769b66b-86k92\" (UID: \"9c0394d2-076a-4bef-9198-be9301946e20\") " pod="openshift-monitoring/metrics-server-5f5769b66b-86k92" Jul 06 19:29:05.774512 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:05.774427 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9c0394d2-076a-4bef-9198-be9301946e20-secret-metrics-server-tls\") pod \"metrics-server-5f5769b66b-86k92\" (UID: \"9c0394d2-076a-4bef-9198-be9301946e20\") " pod="openshift-monitoring/metrics-server-5f5769b66b-86k92" Jul 06 19:29:05.774512 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:05.774472 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9c0394d2-076a-4bef-9198-be9301946e20-secret-metrics-server-client-certs\") pod \"metrics-server-5f5769b66b-86k92\" (UID: \"9c0394d2-076a-4bef-9198-be9301946e20\") " pod="openshift-monitoring/metrics-server-5f5769b66b-86k92" Jul 06 19:29:05.774605 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:05.774523 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9c0394d2-076a-4bef-9198-be9301946e20-client-ca-bundle\") pod \"metrics-server-5f5769b66b-86k92\" (UID: \"9c0394d2-076a-4bef-9198-be9301946e20\") " pod="openshift-monitoring/metrics-server-5f5769b66b-86k92" Jul 06 19:29:05.875309 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:05.875265 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9c0394d2-076a-4bef-9198-be9301946e20-metrics-server-audit-profiles\") pod \"metrics-server-5f5769b66b-86k92\" (UID: \"9c0394d2-076a-4bef-9198-be9301946e20\") " pod="openshift-monitoring/metrics-server-5f5769b66b-86k92" Jul 06 19:29:05.875309 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:05.875311 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9c0394d2-076a-4bef-9198-be9301946e20-audit-log\") pod \"metrics-server-5f5769b66b-86k92\" (UID: \"9c0394d2-076a-4bef-9198-be9301946e20\") " pod="openshift-monitoring/metrics-server-5f5769b66b-86k92" Jul 06 19:29:05.875518 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:05.875447 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9c0394d2-076a-4bef-9198-be9301946e20-secret-metrics-server-tls\") pod \"metrics-server-5f5769b66b-86k92\" (UID: \"9c0394d2-076a-4bef-9198-be9301946e20\") " pod="openshift-monitoring/metrics-server-5f5769b66b-86k92" Jul 06 19:29:05.875518 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:05.875487 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9c0394d2-076a-4bef-9198-be9301946e20-secret-metrics-server-client-certs\") pod \"metrics-server-5f5769b66b-86k92\" (UID: \"9c0394d2-076a-4bef-9198-be9301946e20\") " pod="openshift-monitoring/metrics-server-5f5769b66b-86k92" Jul 06 19:29:05.875621 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:05.875545 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9c0394d2-076a-4bef-9198-be9301946e20-client-ca-bundle\") pod \"metrics-server-5f5769b66b-86k92\" (UID: \"9c0394d2-076a-4bef-9198-be9301946e20\") " pod="openshift-monitoring/metrics-server-5f5769b66b-86k92" Jul 06 19:29:05.875621 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:05.875595 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2zlf\" (UniqueName: \"kubernetes.io/projected/9c0394d2-076a-4bef-9198-be9301946e20-kube-api-access-q2zlf\") pod \"metrics-server-5f5769b66b-86k92\" (UID: \"9c0394d2-076a-4bef-9198-be9301946e20\") " pod="openshift-monitoring/metrics-server-5f5769b66b-86k92" Jul 06 19:29:05.875720 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:05.875621 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c0394d2-076a-4bef-9198-be9301946e20-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f5769b66b-86k92\" (UID: \"9c0394d2-076a-4bef-9198-be9301946e20\") " pod="openshift-monitoring/metrics-server-5f5769b66b-86k92" Jul 06 19:29:05.875720 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:05.875679 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9c0394d2-076a-4bef-9198-be9301946e20-audit-log\") pod \"metrics-server-5f5769b66b-86k92\" (UID: \"9c0394d2-076a-4bef-9198-be9301946e20\") " pod="openshift-monitoring/metrics-server-5f5769b66b-86k92" Jul 06 19:29:05.876506 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:05.876475 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9c0394d2-076a-4bef-9198-be9301946e20-metrics-server-audit-profiles\") pod \"metrics-server-5f5769b66b-86k92\" (UID: \"9c0394d2-076a-4bef-9198-be9301946e20\") " pod="openshift-monitoring/metrics-server-5f5769b66b-86k92" Jul 06 19:29:05.876882 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:05.876821 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c0394d2-076a-4bef-9198-be9301946e20-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f5769b66b-86k92\" (UID: \"9c0394d2-076a-4bef-9198-be9301946e20\") " pod="openshift-monitoring/metrics-server-5f5769b66b-86k92" Jul 06 19:29:05.879049 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:05.879027 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9c0394d2-076a-4bef-9198-be9301946e20-secret-metrics-server-client-certs\") pod \"metrics-server-5f5769b66b-86k92\" (UID: \"9c0394d2-076a-4bef-9198-be9301946e20\") " pod="openshift-monitoring/metrics-server-5f5769b66b-86k92" Jul 06 19:29:05.879524 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:05.879504 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9c0394d2-076a-4bef-9198-be9301946e20-client-ca-bundle\") pod \"metrics-server-5f5769b66b-86k92\" (UID: \"9c0394d2-076a-4bef-9198-be9301946e20\") " pod="openshift-monitoring/metrics-server-5f5769b66b-86k92" Jul 06 19:29:05.879956 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:05.879915 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9c0394d2-076a-4bef-9198-be9301946e20-secret-metrics-server-tls\") pod \"metrics-server-5f5769b66b-86k92\" (UID: \"9c0394d2-076a-4bef-9198-be9301946e20\") " pod="openshift-monitoring/metrics-server-5f5769b66b-86k92" Jul 06 19:29:05.885278 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:05.885239 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2zlf\" (UniqueName: \"kubernetes.io/projected/9c0394d2-076a-4bef-9198-be9301946e20-kube-api-access-q2zlf\") pod \"metrics-server-5f5769b66b-86k92\" (UID: \"9c0394d2-076a-4bef-9198-be9301946e20\") " pod="openshift-monitoring/metrics-server-5f5769b66b-86k92" Jul 06 19:29:06.071385 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.071203 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5f5769b66b-86k92" Jul 06 19:29:06.263005 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.262969 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d74c98c7c-kzskh"] Jul 06 19:29:06.266409 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.266385 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d74c98c7c-kzskh" Jul 06 19:29:06.274166 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.274143 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 06 19:29:06.280018 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.279995 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d74c98c7c-kzskh"] Jul 06 19:29:06.280309 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.280293 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa5d74e3-4974-4e1f-bbfd-4ffc679e7780-service-ca\") pod \"console-5d74c98c7c-kzskh\" (UID: \"aa5d74e3-4974-4e1f-bbfd-4ffc679e7780\") " pod="openshift-console/console-5d74c98c7c-kzskh" Jul 06 19:29:06.280386 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.280325 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa5d74e3-4974-4e1f-bbfd-4ffc679e7780-console-config\") pod \"console-5d74c98c7c-kzskh\" (UID: \"aa5d74e3-4974-4e1f-bbfd-4ffc679e7780\") " pod="openshift-console/console-5d74c98c7c-kzskh" Jul 06 19:29:06.280386 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.280363 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa5d74e3-4974-4e1f-bbfd-4ffc679e7780-trusted-ca-bundle\") pod \"console-5d74c98c7c-kzskh\" (UID: \"aa5d74e3-4974-4e1f-bbfd-4ffc679e7780\") " pod="openshift-console/console-5d74c98c7c-kzskh" Jul 06 19:29:06.280489 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.280390 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa5d74e3-4974-4e1f-bbfd-4ffc679e7780-console-oauth-config\") pod \"console-5d74c98c7c-kzskh\" (UID: \"aa5d74e3-4974-4e1f-bbfd-4ffc679e7780\") " pod="openshift-console/console-5d74c98c7c-kzskh" Jul 06 19:29:06.280489 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.280424 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs5jj\" (UniqueName: \"kubernetes.io/projected/aa5d74e3-4974-4e1f-bbfd-4ffc679e7780-kube-api-access-vs5jj\") pod \"console-5d74c98c7c-kzskh\" (UID: \"aa5d74e3-4974-4e1f-bbfd-4ffc679e7780\") " pod="openshift-console/console-5d74c98c7c-kzskh" Jul 06 19:29:06.280571 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.280515 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa5d74e3-4974-4e1f-bbfd-4ffc679e7780-console-serving-cert\") pod \"console-5d74c98c7c-kzskh\" (UID: \"aa5d74e3-4974-4e1f-bbfd-4ffc679e7780\") " pod="openshift-console/console-5d74c98c7c-kzskh" Jul 06 19:29:06.280571 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.280557 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa5d74e3-4974-4e1f-bbfd-4ffc679e7780-oauth-serving-cert\") pod \"console-5d74c98c7c-kzskh\" (UID: \"aa5d74e3-4974-4e1f-bbfd-4ffc679e7780\") " pod="openshift-console/console-5d74c98c7c-kzskh" Jul 06 19:29:06.362162 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.362089 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6749c8c9bc-x59nv"] Jul 06 19:29:06.365468 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.365450 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6749c8c9bc-x59nv" Jul 06 19:29:06.368311 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.368288 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 06 19:29:06.368446 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.368427 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 06 19:29:06.368615 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.368600 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 06 19:29:06.368836 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.368700 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 06 19:29:06.368933 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.368908 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 06 19:29:06.368986 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.368955 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-s7wqn\"" Jul 06 19:29:06.379520 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.379495 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 06 19:29:06.380462 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.380444 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6749c8c9bc-x59nv"] Jul 06 19:29:06.381112 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.381076 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa5d74e3-4974-4e1f-bbfd-4ffc679e7780-console-oauth-config\") pod \"console-5d74c98c7c-kzskh\" (UID: \"aa5d74e3-4974-4e1f-bbfd-4ffc679e7780\") " pod="openshift-console/console-5d74c98c7c-kzskh" Jul 06 19:29:06.381112 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.381117 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs5jj\" (UniqueName: \"kubernetes.io/projected/aa5d74e3-4974-4e1f-bbfd-4ffc679e7780-kube-api-access-vs5jj\") pod \"console-5d74c98c7c-kzskh\" (UID: \"aa5d74e3-4974-4e1f-bbfd-4ffc679e7780\") " pod="openshift-console/console-5d74c98c7c-kzskh" Jul 06 19:29:06.381274 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.381144 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15a01f49-5b2a-4c49-8f3b-4f23a6d48b45-serving-certs-ca-bundle\") pod \"telemeter-client-6749c8c9bc-x59nv\" (UID: \"15a01f49-5b2a-4c49-8f3b-4f23a6d48b45\") " pod="openshift-monitoring/telemeter-client-6749c8c9bc-x59nv" Jul 06 19:29:06.381274 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.381202 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/15a01f49-5b2a-4c49-8f3b-4f23a6d48b45-telemeter-client-tls\") pod \"telemeter-client-6749c8c9bc-x59nv\" (UID: \"15a01f49-5b2a-4c49-8f3b-4f23a6d48b45\") " pod="openshift-monitoring/telemeter-client-6749c8c9bc-x59nv" Jul 06 19:29:06.381274 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.381225 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15a01f49-5b2a-4c49-8f3b-4f23a6d48b45-metrics-client-ca\") pod \"telemeter-client-6749c8c9bc-x59nv\" (UID: \"15a01f49-5b2a-4c49-8f3b-4f23a6d48b45\") " pod="openshift-monitoring/telemeter-client-6749c8c9bc-x59nv" Jul 06 19:29:06.381274 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.381256 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/15a01f49-5b2a-4c49-8f3b-4f23a6d48b45-federate-client-tls\") pod \"telemeter-client-6749c8c9bc-x59nv\" (UID: \"15a01f49-5b2a-4c49-8f3b-4f23a6d48b45\") " pod="openshift-monitoring/telemeter-client-6749c8c9bc-x59nv" Jul 06 19:29:06.381469 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.381333 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15a01f49-5b2a-4c49-8f3b-4f23a6d48b45-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6749c8c9bc-x59nv\" (UID: \"15a01f49-5b2a-4c49-8f3b-4f23a6d48b45\") " pod="openshift-monitoring/telemeter-client-6749c8c9bc-x59nv" Jul 06 19:29:06.381469 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.381396 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa5d74e3-4974-4e1f-bbfd-4ffc679e7780-console-serving-cert\") pod \"console-5d74c98c7c-kzskh\" (UID: \"aa5d74e3-4974-4e1f-bbfd-4ffc679e7780\") " pod="openshift-console/console-5d74c98c7c-kzskh" Jul 06 19:29:06.381469 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.381430 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa5d74e3-4974-4e1f-bbfd-4ffc679e7780-oauth-serving-cert\") pod \"console-5d74c98c7c-kzskh\" (UID: \"aa5d74e3-4974-4e1f-bbfd-4ffc679e7780\") " pod="openshift-console/console-5d74c98c7c-kzskh" Jul 06 19:29:06.381469 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.381456 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/15a01f49-5b2a-4c49-8f3b-4f23a6d48b45-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6749c8c9bc-x59nv\" (UID: \"15a01f49-5b2a-4c49-8f3b-4f23a6d48b45\") " pod="openshift-monitoring/telemeter-client-6749c8c9bc-x59nv" Jul 06 19:29:06.381654 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.381482 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/15a01f49-5b2a-4c49-8f3b-4f23a6d48b45-secret-telemeter-client\") pod \"telemeter-client-6749c8c9bc-x59nv\" (UID: \"15a01f49-5b2a-4c49-8f3b-4f23a6d48b45\") " pod="openshift-monitoring/telemeter-client-6749c8c9bc-x59nv" Jul 06 19:29:06.381654 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.381508 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wzdh\" (UniqueName: \"kubernetes.io/projected/15a01f49-5b2a-4c49-8f3b-4f23a6d48b45-kube-api-access-7wzdh\") pod \"telemeter-client-6749c8c9bc-x59nv\" (UID: \"15a01f49-5b2a-4c49-8f3b-4f23a6d48b45\") " pod="openshift-monitoring/telemeter-client-6749c8c9bc-x59nv" Jul 06 19:29:06.381654 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.381544 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa5d74e3-4974-4e1f-bbfd-4ffc679e7780-service-ca\") pod \"console-5d74c98c7c-kzskh\" (UID: \"aa5d74e3-4974-4e1f-bbfd-4ffc679e7780\") " pod="openshift-console/console-5d74c98c7c-kzskh" Jul 06 19:29:06.381654 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.381627 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa5d74e3-4974-4e1f-bbfd-4ffc679e7780-console-config\") pod \"console-5d74c98c7c-kzskh\" (UID: \"aa5d74e3-4974-4e1f-bbfd-4ffc679e7780\") " pod="openshift-console/console-5d74c98c7c-kzskh" Jul 06 19:29:06.381881 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.381679 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa5d74e3-4974-4e1f-bbfd-4ffc679e7780-trusted-ca-bundle\") pod \"console-5d74c98c7c-kzskh\" (UID: \"aa5d74e3-4974-4e1f-bbfd-4ffc679e7780\") " pod="openshift-console/console-5d74c98c7c-kzskh" Jul 06 19:29:06.382141 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.382117 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa5d74e3-4974-4e1f-bbfd-4ffc679e7780-oauth-serving-cert\") pod \"console-5d74c98c7c-kzskh\" (UID: \"aa5d74e3-4974-4e1f-bbfd-4ffc679e7780\") " pod="openshift-console/console-5d74c98c7c-kzskh" Jul 06 19:29:06.382747 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.382706 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa5d74e3-4974-4e1f-bbfd-4ffc679e7780-service-ca\") pod \"console-5d74c98c7c-kzskh\" (UID: \"aa5d74e3-4974-4e1f-bbfd-4ffc679e7780\") " pod="openshift-console/console-5d74c98c7c-kzskh" Jul 06 19:29:06.383631 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.383612 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa5d74e3-4974-4e1f-bbfd-4ffc679e7780-trusted-ca-bundle\") pod \"console-5d74c98c7c-kzskh\" (UID: \"aa5d74e3-4974-4e1f-bbfd-4ffc679e7780\") " pod="openshift-console/console-5d74c98c7c-kzskh" Jul 06 19:29:06.383795 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.383644 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa5d74e3-4974-4e1f-bbfd-4ffc679e7780-console-config\") pod \"console-5d74c98c7c-kzskh\" (UID: \"aa5d74e3-4974-4e1f-bbfd-4ffc679e7780\") " pod="openshift-console/console-5d74c98c7c-kzskh" Jul 06 19:29:06.384161 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.384122 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa5d74e3-4974-4e1f-bbfd-4ffc679e7780-console-oauth-config\") pod \"console-5d74c98c7c-kzskh\" (UID: \"aa5d74e3-4974-4e1f-bbfd-4ffc679e7780\") " pod="openshift-console/console-5d74c98c7c-kzskh" Jul 06 19:29:06.384731 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.384711 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa5d74e3-4974-4e1f-bbfd-4ffc679e7780-console-serving-cert\") pod \"console-5d74c98c7c-kzskh\" (UID: \"aa5d74e3-4974-4e1f-bbfd-4ffc679e7780\") " pod="openshift-console/console-5d74c98c7c-kzskh" Jul 06 19:29:06.394134 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.394111 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs5jj\" (UniqueName: \"kubernetes.io/projected/aa5d74e3-4974-4e1f-bbfd-4ffc679e7780-kube-api-access-vs5jj\") pod \"console-5d74c98c7c-kzskh\" (UID: \"aa5d74e3-4974-4e1f-bbfd-4ffc679e7780\") " pod="openshift-console/console-5d74c98c7c-kzskh" Jul 06 19:29:06.482065 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.482036 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15a01f49-5b2a-4c49-8f3b-4f23a6d48b45-serving-certs-ca-bundle\") pod \"telemeter-client-6749c8c9bc-x59nv\" (UID: \"15a01f49-5b2a-4c49-8f3b-4f23a6d48b45\") " pod="openshift-monitoring/telemeter-client-6749c8c9bc-x59nv" Jul 06 19:29:06.482225 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.482078 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/15a01f49-5b2a-4c49-8f3b-4f23a6d48b45-telemeter-client-tls\") pod \"telemeter-client-6749c8c9bc-x59nv\" (UID: \"15a01f49-5b2a-4c49-8f3b-4f23a6d48b45\") " pod="openshift-monitoring/telemeter-client-6749c8c9bc-x59nv" Jul 06 19:29:06.482225 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.482095 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15a01f49-5b2a-4c49-8f3b-4f23a6d48b45-metrics-client-ca\") pod \"telemeter-client-6749c8c9bc-x59nv\" (UID: \"15a01f49-5b2a-4c49-8f3b-4f23a6d48b45\") " pod="openshift-monitoring/telemeter-client-6749c8c9bc-x59nv" Jul 06 19:29:06.482225 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.482176 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/15a01f49-5b2a-4c49-8f3b-4f23a6d48b45-federate-client-tls\") pod \"telemeter-client-6749c8c9bc-x59nv\" (UID: \"15a01f49-5b2a-4c49-8f3b-4f23a6d48b45\") " pod="openshift-monitoring/telemeter-client-6749c8c9bc-x59nv" Jul 06 19:29:06.482391 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.482356 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15a01f49-5b2a-4c49-8f3b-4f23a6d48b45-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6749c8c9bc-x59nv\" (UID: \"15a01f49-5b2a-4c49-8f3b-4f23a6d48b45\") " pod="openshift-monitoring/telemeter-client-6749c8c9bc-x59nv" Jul 06 19:29:06.482442 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.482415 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/15a01f49-5b2a-4c49-8f3b-4f23a6d48b45-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6749c8c9bc-x59nv\" (UID: \"15a01f49-5b2a-4c49-8f3b-4f23a6d48b45\") " pod="openshift-monitoring/telemeter-client-6749c8c9bc-x59nv" Jul 06 19:29:06.482494 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.482444 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/15a01f49-5b2a-4c49-8f3b-4f23a6d48b45-secret-telemeter-client\") pod \"telemeter-client-6749c8c9bc-x59nv\" (UID: \"15a01f49-5b2a-4c49-8f3b-4f23a6d48b45\") " pod="openshift-monitoring/telemeter-client-6749c8c9bc-x59nv" Jul 06 19:29:06.482494 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.482470 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wzdh\" (UniqueName: \"kubernetes.io/projected/15a01f49-5b2a-4c49-8f3b-4f23a6d48b45-kube-api-access-7wzdh\") pod \"telemeter-client-6749c8c9bc-x59nv\" (UID: \"15a01f49-5b2a-4c49-8f3b-4f23a6d48b45\") " pod="openshift-monitoring/telemeter-client-6749c8c9bc-x59nv" Jul 06 19:29:06.482845 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.482818 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15a01f49-5b2a-4c49-8f3b-4f23a6d48b45-serving-certs-ca-bundle\") pod \"telemeter-client-6749c8c9bc-x59nv\" (UID: \"15a01f49-5b2a-4c49-8f3b-4f23a6d48b45\") " pod="openshift-monitoring/telemeter-client-6749c8c9bc-x59nv" Jul 06 19:29:06.482941 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.482845 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15a01f49-5b2a-4c49-8f3b-4f23a6d48b45-metrics-client-ca\") pod \"telemeter-client-6749c8c9bc-x59nv\" (UID: \"15a01f49-5b2a-4c49-8f3b-4f23a6d48b45\") " pod="openshift-monitoring/telemeter-client-6749c8c9bc-x59nv" Jul 06 19:29:06.483625 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.483603 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15a01f49-5b2a-4c49-8f3b-4f23a6d48b45-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6749c8c9bc-x59nv\" (UID: \"15a01f49-5b2a-4c49-8f3b-4f23a6d48b45\") " pod="openshift-monitoring/telemeter-client-6749c8c9bc-x59nv" Jul 06 19:29:06.484967 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.484943 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/15a01f49-5b2a-4c49-8f3b-4f23a6d48b45-telemeter-client-tls\") pod \"telemeter-client-6749c8c9bc-x59nv\" (UID: \"15a01f49-5b2a-4c49-8f3b-4f23a6d48b45\") " pod="openshift-monitoring/telemeter-client-6749c8c9bc-x59nv" Jul 06 19:29:06.485087 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.485052 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/15a01f49-5b2a-4c49-8f3b-4f23a6d48b45-federate-client-tls\") pod \"telemeter-client-6749c8c9bc-x59nv\" (UID: \"15a01f49-5b2a-4c49-8f3b-4f23a6d48b45\") " pod="openshift-monitoring/telemeter-client-6749c8c9bc-x59nv" Jul 06 19:29:06.485295 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.485277 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/15a01f49-5b2a-4c49-8f3b-4f23a6d48b45-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6749c8c9bc-x59nv\" (UID: \"15a01f49-5b2a-4c49-8f3b-4f23a6d48b45\") " pod="openshift-monitoring/telemeter-client-6749c8c9bc-x59nv" Jul 06 19:29:06.485505 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.485487 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/15a01f49-5b2a-4c49-8f3b-4f23a6d48b45-secret-telemeter-client\") pod \"telemeter-client-6749c8c9bc-x59nv\" (UID: \"15a01f49-5b2a-4c49-8f3b-4f23a6d48b45\") " pod="openshift-monitoring/telemeter-client-6749c8c9bc-x59nv" Jul 06 19:29:06.492448 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.492429 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wzdh\" (UniqueName: \"kubernetes.io/projected/15a01f49-5b2a-4c49-8f3b-4f23a6d48b45-kube-api-access-7wzdh\") pod \"telemeter-client-6749c8c9bc-x59nv\" (UID: \"15a01f49-5b2a-4c49-8f3b-4f23a6d48b45\") " pod="openshift-monitoring/telemeter-client-6749c8c9bc-x59nv" Jul 06 19:29:06.578237 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.578201 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d74c98c7c-kzskh" Jul 06 19:29:06.677771 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:06.677700 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6749c8c9bc-x59nv" Jul 06 19:29:08.993631 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:29:08.993599 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-272x4" podUID="1adc5a22-7a57-4da4-9a66-0c451f2603be" Jul 06 19:29:09.405904 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:09.405868 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-272x4" Jul 06 19:29:13.589729 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:13.589707 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6749c8c9bc-x59nv"] Jul 06 19:29:13.592803 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:29:13.592766 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15a01f49_5b2a_4c49_8f3b_4f23a6d48b45.slice/crio-d11885d9eb26eef5f9d71493ceec40aa10cc9527c82fd284212b10e6fdbd6f68 WatchSource:0}: Error finding container d11885d9eb26eef5f9d71493ceec40aa10cc9527c82fd284212b10e6fdbd6f68: Status 404 returned error can't find the container with id d11885d9eb26eef5f9d71493ceec40aa10cc9527c82fd284212b10e6fdbd6f68 Jul 06 19:29:13.603038 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:13.602522 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d74c98c7c-kzskh"] Jul 06 19:29:13.605793 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:29:13.605768 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa5d74e3_4974_4e1f_bbfd_4ffc679e7780.slice/crio-209bbfd4239b964f5710befab51db12284a66e370cbc90bc7f2386e98169cab8 WatchSource:0}: Error finding container 209bbfd4239b964f5710befab51db12284a66e370cbc90bc7f2386e98169cab8: Status 404 returned error can't find the container with id 209bbfd4239b964f5710befab51db12284a66e370cbc90bc7f2386e98169cab8 Jul 06 19:29:13.620036 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:13.620012 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5f5769b66b-86k92"] Jul 06 19:29:13.622518 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:29:13.622485 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c0394d2_076a_4bef_9198_be9301946e20.slice/crio-a06ea598492bfca46949efb099bff018a5e5c7b91eb08f1da45e72d675bc12c8 WatchSource:0}: Error finding container a06ea598492bfca46949efb099bff018a5e5c7b91eb08f1da45e72d675bc12c8: Status 404 returned error can't find the container with id a06ea598492bfca46949efb099bff018a5e5c7b91eb08f1da45e72d675bc12c8 Jul 06 19:29:14.045377 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:14.045287 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1adc5a22-7a57-4da4-9a66-0c451f2603be-metrics-tls\") pod \"dns-default-272x4\" (UID: \"1adc5a22-7a57-4da4-9a66-0c451f2603be\") " pod="openshift-dns/dns-default-272x4" Jul 06 19:29:14.047976 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:14.047950 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1adc5a22-7a57-4da4-9a66-0c451f2603be-metrics-tls\") pod \"dns-default-272x4\" (UID: \"1adc5a22-7a57-4da4-9a66-0c451f2603be\") " pod="openshift-dns/dns-default-272x4" Jul 06 19:29:14.209167 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:14.209135 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7f4qs\"" Jul 06 19:29:14.216957 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:14.216915 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-272x4" Jul 06 19:29:14.399210 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:14.399147 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-272x4"] Jul 06 19:29:14.409981 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:29:14.409946 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1adc5a22_7a57_4da4_9a66_0c451f2603be.slice/crio-9ce3e82767c3ec57bac7b23db010885d99bc9594f1df7bafd1a19b4f3f58e4a1 WatchSource:0}: Error finding container 9ce3e82767c3ec57bac7b23db010885d99bc9594f1df7bafd1a19b4f3f58e4a1: Status 404 returned error can't find the container with id 9ce3e82767c3ec57bac7b23db010885d99bc9594f1df7bafd1a19b4f3f58e4a1 Jul 06 19:29:14.430651 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:14.430582 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6749c8c9bc-x59nv" event={"ID":"15a01f49-5b2a-4c49-8f3b-4f23a6d48b45","Type":"ContainerStarted","Data":"d11885d9eb26eef5f9d71493ceec40aa10cc9527c82fd284212b10e6fdbd6f68"} Jul 06 19:29:14.433423 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:14.433092 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-rsg5c" event={"ID":"d3f27b98-edb6-4774-a5ea-330b55f2e0ec","Type":"ContainerStarted","Data":"62413086d50b5076f1b8cc45729717d4e4505c7896f3033070ebeb87db5324f3"} Jul 06 19:29:14.433705 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:14.433646 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-rsg5c" Jul 06 19:29:14.437306 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:14.436928 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c7cf86bbd-ck249" event={"ID":"cf8e44b4-157b-43f6-b854-5b39e1e718c9","Type":"ContainerStarted","Data":"40e512d34b0338c6cdebf69fb15f6eaa740a97d3f7c33b2e636a53e802164315"} Jul 06 19:29:14.437306 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:14.437003 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c7cf86bbd-ck249" event={"ID":"cf8e44b4-157b-43f6-b854-5b39e1e718c9","Type":"ContainerStarted","Data":"07f788ab22e9264cf963ad288f058ae3a11a6f58c19bd6e976e7d0ce52010aa2"} Jul 06 19:29:14.437306 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:14.437047 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c7cf86bbd-ck249" event={"ID":"cf8e44b4-157b-43f6-b854-5b39e1e718c9","Type":"ContainerStarted","Data":"55dba3790af947ae240b359d4200619f6453fc7ab11d5a824d3bec818a7ea148"} Jul 06 19:29:14.438575 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:14.438550 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5f5769b66b-86k92" event={"ID":"9c0394d2-076a-4bef-9198-be9301946e20","Type":"ContainerStarted","Data":"a06ea598492bfca46949efb099bff018a5e5c7b91eb08f1da45e72d675bc12c8"} Jul 06 19:29:14.440604 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:14.440577 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-272x4" event={"ID":"1adc5a22-7a57-4da4-9a66-0c451f2603be","Type":"ContainerStarted","Data":"9ce3e82767c3ec57bac7b23db010885d99bc9594f1df7bafd1a19b4f3f58e4a1"} Jul 06 19:29:14.443404 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:14.443375 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d74c98c7c-kzskh" event={"ID":"aa5d74e3-4974-4e1f-bbfd-4ffc679e7780","Type":"ContainerStarted","Data":"7fe864f176a976b0a923726acc9f1a9758a329b415a5b941562d69c495619079"} Jul 06 19:29:14.443502 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:14.443421 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d74c98c7c-kzskh" event={"ID":"aa5d74e3-4974-4e1f-bbfd-4ffc679e7780","Type":"ContainerStarted","Data":"209bbfd4239b964f5710befab51db12284a66e370cbc90bc7f2386e98169cab8"} Jul 06 19:29:14.448355 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:14.448339 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-rsg5c" Jul 06 19:29:14.453343 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:14.453289 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-rsg5c" podStartSLOduration=2.315467262 podStartE2EDuration="22.453273003s" totalImagesPullingTime="20.137805741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:28:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:29:14.452290407 +0000 UTC m=+163.272379142" watchObservedRunningTime="2026-07-06 19:29:14.453273003 +0000 UTC m=+163.273361715" Jul 06 19:29:16.578680 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:16.578642 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d74c98c7c-kzskh" Jul 06 19:29:16.579077 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:16.578718 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d74c98c7c-kzskh" Jul 06 19:29:16.583992 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:16.583971 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d74c98c7c-kzskh" Jul 06 19:29:16.623775 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:16.623722 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d74c98c7c-kzskh" podStartSLOduration=10.623703866 podStartE2EDuration="10.623703866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:29:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:29:14.508256269 +0000 UTC m=+163.328344980" watchObservedRunningTime="2026-07-06 19:29:16.623703866 +0000 UTC m=+165.443792580" Jul 06 19:29:17.466049 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:17.466021 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d74c98c7c-kzskh" Jul 06 19:29:17.590462 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:17.590433 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bf496c95-7q6sp"] Jul 06 19:29:18.467705 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:18.467569 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-272x4" event={"ID":"1adc5a22-7a57-4da4-9a66-0c451f2603be","Type":"ContainerStarted","Data":"0d4b2ba6b789f7f4dc661c5181fee07739f1d657ae5c21bacb81fe2ae36c0bea"} Jul 06 19:29:18.467705 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:18.467641 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-272x4" event={"ID":"1adc5a22-7a57-4da4-9a66-0c451f2603be","Type":"ContainerStarted","Data":"9bc2cb332d92e922c7d185fa6e5bfd1cc834f292c71494b2b2ecdc8dad3104dd"} Jul 06 19:29:18.467705 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:18.467686 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-272x4" Jul 06 19:29:18.469232 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:18.469205 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6749c8c9bc-x59nv" event={"ID":"15a01f49-5b2a-4c49-8f3b-4f23a6d48b45","Type":"ContainerStarted","Data":"837aa899ec839a65f16cc96e084e55fb2f8006434a44d673f9996d4fe433d7f5"} Jul 06 19:29:18.473307 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:18.473285 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c7cf86bbd-ck249" event={"ID":"cf8e44b4-157b-43f6-b854-5b39e1e718c9","Type":"ContainerStarted","Data":"bb46d477c0f58694c75dd09a767f8892646b0eb689e9f465faa79d64a694bfd7"} Jul 06 19:29:18.473449 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:18.473429 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c7cf86bbd-ck249" event={"ID":"cf8e44b4-157b-43f6-b854-5b39e1e718c9","Type":"ContainerStarted","Data":"6235c84a6b27bdc5438f3249d0a1f07097dd4eaa1ad8461e6cf5c955bb6a7976"} Jul 06 19:29:18.473525 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:18.473483 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7c7cf86bbd-ck249" Jul 06 19:29:18.473577 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:18.473531 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c7cf86bbd-ck249" event={"ID":"cf8e44b4-157b-43f6-b854-5b39e1e718c9","Type":"ContainerStarted","Data":"cb1fe2689c91fcf633e2175d41429210f66ab2b4fc8cc787cb98b83db97a9776"} Jul 06 19:29:18.476062 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:18.476016 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5f5769b66b-86k92" event={"ID":"9c0394d2-076a-4bef-9198-be9301946e20","Type":"ContainerStarted","Data":"d1caa20876d765a6334087991e617bc77af358bd3e39ca439ae9cd4f7dc888aa"} Jul 06 19:29:18.479823 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:18.479797 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7c7cf86bbd-ck249" Jul 06 19:29:18.504374 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:18.504296 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-272x4" podStartSLOduration=129.856377709 podStartE2EDuration="2m13.50428365s" totalImagesPullingTime="3.647905941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:27:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:29:18.501459147 +0000 UTC m=+167.321547855" watchObservedRunningTime="2026-07-06 19:29:18.50428365 +0000 UTC m=+167.324372342" Jul 06 19:29:18.540956 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:18.540909 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7c7cf86bbd-ck249" podStartSLOduration=2.075727157 podStartE2EDuration="14.54089686s" totalImagesPullingTime="12.465169703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:29:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:29:18.540703359 +0000 UTC m=+167.360792071" watchObservedRunningTime="2026-07-06 19:29:18.54089686 +0000 UTC m=+167.360985572" Jul 06 19:29:18.569637 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:18.569589 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5f5769b66b-86k92" podStartSLOduration=9.6033578 podStartE2EDuration="13.569570688s" totalImagesPullingTime="3.966212888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:29:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:29:18.567474902 +0000 UTC m=+167.387563637" watchObservedRunningTime="2026-07-06 19:29:18.569570688 +0000 UTC m=+167.389659428" Jul 06 19:29:20.489412 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:20.489368 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6749c8c9bc-x59nv" event={"ID":"15a01f49-5b2a-4c49-8f3b-4f23a6d48b45","Type":"ContainerStarted","Data":"1c5561dc597765273643a882aab4983dcb0367fcc405d53b3fe15b05854c3987"} Jul 06 19:29:20.489986 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:20.489445 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6749c8c9bc-x59nv" event={"ID":"15a01f49-5b2a-4c49-8f3b-4f23a6d48b45","Type":"ContainerStarted","Data":"9424137dd3d61457bd9e0e36f4d6498b9dfeead07fdddfdee7e693e76a4c79b5"} Jul 06 19:29:20.525063 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:20.524976 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6749c8c9bc-x59nv" podStartSLOduration=8.816535898 podStartE2EDuration="14.524956653s" totalImagesPullingTime="5.708420755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:29:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:29:20.523086139 +0000 UTC m=+169.343174864" watchObservedRunningTime="2026-07-06 19:29:20.524956653 +0000 UTC m=+169.345045365" Jul 06 19:29:21.467168 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:21.467132 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-c7f5bdd96-tkv4l"] Jul 06 19:29:21.501671 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:21.501643 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c7f5bdd96-tkv4l" Jul 06 19:29:21.506602 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:21.506575 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c7f5bdd96-tkv4l"] Jul 06 19:29:21.620573 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:21.620537 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c8d6b42-b114-44e7-bc29-4f1552633e17-trusted-ca-bundle\") pod \"console-c7f5bdd96-tkv4l\" (UID: \"2c8d6b42-b114-44e7-bc29-4f1552633e17\") " pod="openshift-console/console-c7f5bdd96-tkv4l" Jul 06 19:29:21.620755 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:21.620579 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxlnl\" (UniqueName: \"kubernetes.io/projected/2c8d6b42-b114-44e7-bc29-4f1552633e17-kube-api-access-xxlnl\") pod \"console-c7f5bdd96-tkv4l\" (UID: \"2c8d6b42-b114-44e7-bc29-4f1552633e17\") " pod="openshift-console/console-c7f5bdd96-tkv4l" Jul 06 19:29:21.620755 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:21.620615 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2c8d6b42-b114-44e7-bc29-4f1552633e17-oauth-serving-cert\") pod \"console-c7f5bdd96-tkv4l\" (UID: \"2c8d6b42-b114-44e7-bc29-4f1552633e17\") " pod="openshift-console/console-c7f5bdd96-tkv4l" Jul 06 19:29:21.620755 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:21.620651 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2c8d6b42-b114-44e7-bc29-4f1552633e17-console-serving-cert\") pod \"console-c7f5bdd96-tkv4l\" (UID: \"2c8d6b42-b114-44e7-bc29-4f1552633e17\") " pod="openshift-console/console-c7f5bdd96-tkv4l" Jul 06 19:29:21.621052 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:21.621018 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2c8d6b42-b114-44e7-bc29-4f1552633e17-service-ca\") pod \"console-c7f5bdd96-tkv4l\" (UID: \"2c8d6b42-b114-44e7-bc29-4f1552633e17\") " pod="openshift-console/console-c7f5bdd96-tkv4l" Jul 06 19:29:21.621200 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:21.621084 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2c8d6b42-b114-44e7-bc29-4f1552633e17-console-config\") pod \"console-c7f5bdd96-tkv4l\" (UID: \"2c8d6b42-b114-44e7-bc29-4f1552633e17\") " pod="openshift-console/console-c7f5bdd96-tkv4l" Jul 06 19:29:21.621200 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:21.621121 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2c8d6b42-b114-44e7-bc29-4f1552633e17-console-oauth-config\") pod \"console-c7f5bdd96-tkv4l\" (UID: \"2c8d6b42-b114-44e7-bc29-4f1552633e17\") " pod="openshift-console/console-c7f5bdd96-tkv4l" Jul 06 19:29:21.722141 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:21.722059 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2c8d6b42-b114-44e7-bc29-4f1552633e17-service-ca\") pod \"console-c7f5bdd96-tkv4l\" (UID: \"2c8d6b42-b114-44e7-bc29-4f1552633e17\") " pod="openshift-console/console-c7f5bdd96-tkv4l" Jul 06 19:29:21.722141 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:21.722107 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2c8d6b42-b114-44e7-bc29-4f1552633e17-console-config\") pod \"console-c7f5bdd96-tkv4l\" (UID: \"2c8d6b42-b114-44e7-bc29-4f1552633e17\") " pod="openshift-console/console-c7f5bdd96-tkv4l" Jul 06 19:29:21.722141 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:21.722137 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2c8d6b42-b114-44e7-bc29-4f1552633e17-console-oauth-config\") pod \"console-c7f5bdd96-tkv4l\" (UID: \"2c8d6b42-b114-44e7-bc29-4f1552633e17\") " pod="openshift-console/console-c7f5bdd96-tkv4l" Jul 06 19:29:21.722435 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:21.722202 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c8d6b42-b114-44e7-bc29-4f1552633e17-trusted-ca-bundle\") pod \"console-c7f5bdd96-tkv4l\" (UID: \"2c8d6b42-b114-44e7-bc29-4f1552633e17\") " pod="openshift-console/console-c7f5bdd96-tkv4l" Jul 06 19:29:21.722435 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:21.722237 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxlnl\" (UniqueName: \"kubernetes.io/projected/2c8d6b42-b114-44e7-bc29-4f1552633e17-kube-api-access-xxlnl\") pod \"console-c7f5bdd96-tkv4l\" (UID: \"2c8d6b42-b114-44e7-bc29-4f1552633e17\") " pod="openshift-console/console-c7f5bdd96-tkv4l" Jul 06 19:29:21.722435 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:21.722270 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2c8d6b42-b114-44e7-bc29-4f1552633e17-oauth-serving-cert\") pod \"console-c7f5bdd96-tkv4l\" (UID: \"2c8d6b42-b114-44e7-bc29-4f1552633e17\") " pod="openshift-console/console-c7f5bdd96-tkv4l" Jul 06 19:29:21.722435 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:21.722299 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2c8d6b42-b114-44e7-bc29-4f1552633e17-console-serving-cert\") pod \"console-c7f5bdd96-tkv4l\" (UID: \"2c8d6b42-b114-44e7-bc29-4f1552633e17\") " pod="openshift-console/console-c7f5bdd96-tkv4l" Jul 06 19:29:21.723865 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:21.723815 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2c8d6b42-b114-44e7-bc29-4f1552633e17-console-config\") pod \"console-c7f5bdd96-tkv4l\" (UID: \"2c8d6b42-b114-44e7-bc29-4f1552633e17\") " pod="openshift-console/console-c7f5bdd96-tkv4l" Jul 06 19:29:21.723865 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:21.723842 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2c8d6b42-b114-44e7-bc29-4f1552633e17-service-ca\") pod \"console-c7f5bdd96-tkv4l\" (UID: \"2c8d6b42-b114-44e7-bc29-4f1552633e17\") " pod="openshift-console/console-c7f5bdd96-tkv4l" Jul 06 19:29:21.724062 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:21.723993 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c8d6b42-b114-44e7-bc29-4f1552633e17-trusted-ca-bundle\") pod \"console-c7f5bdd96-tkv4l\" (UID: \"2c8d6b42-b114-44e7-bc29-4f1552633e17\") " pod="openshift-console/console-c7f5bdd96-tkv4l" Jul 06 19:29:21.724443 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:21.724419 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2c8d6b42-b114-44e7-bc29-4f1552633e17-oauth-serving-cert\") pod \"console-c7f5bdd96-tkv4l\" (UID: \"2c8d6b42-b114-44e7-bc29-4f1552633e17\") " pod="openshift-console/console-c7f5bdd96-tkv4l" Jul 06 19:29:21.725434 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:21.725412 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2c8d6b42-b114-44e7-bc29-4f1552633e17-console-serving-cert\") pod \"console-c7f5bdd96-tkv4l\" (UID: \"2c8d6b42-b114-44e7-bc29-4f1552633e17\") " pod="openshift-console/console-c7f5bdd96-tkv4l" Jul 06 19:29:21.726558 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:21.726536 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2c8d6b42-b114-44e7-bc29-4f1552633e17-console-oauth-config\") pod \"console-c7f5bdd96-tkv4l\" (UID: \"2c8d6b42-b114-44e7-bc29-4f1552633e17\") " pod="openshift-console/console-c7f5bdd96-tkv4l" Jul 06 19:29:21.733425 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:21.733400 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxlnl\" (UniqueName: \"kubernetes.io/projected/2c8d6b42-b114-44e7-bc29-4f1552633e17-kube-api-access-xxlnl\") pod \"console-c7f5bdd96-tkv4l\" (UID: \"2c8d6b42-b114-44e7-bc29-4f1552633e17\") " pod="openshift-console/console-c7f5bdd96-tkv4l" Jul 06 19:29:21.814483 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:21.814451 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c7f5bdd96-tkv4l" Jul 06 19:29:22.003960 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:22.003880 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c7f5bdd96-tkv4l"] Jul 06 19:29:22.008603 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:29:22.008571 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c8d6b42_b114_44e7_bc29_4f1552633e17.slice/crio-cf8c5d6627d950ff50ba295d027ef11995eb82897ab17aeabdb840db00f17131 WatchSource:0}: Error finding container cf8c5d6627d950ff50ba295d027ef11995eb82897ab17aeabdb840db00f17131: Status 404 returned error can't find the container with id cf8c5d6627d950ff50ba295d027ef11995eb82897ab17aeabdb840db00f17131 Jul 06 19:29:22.499452 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:22.499407 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c7f5bdd96-tkv4l" event={"ID":"2c8d6b42-b114-44e7-bc29-4f1552633e17","Type":"ContainerStarted","Data":"ada61909261528e2e1fb178ef40536ffd000264f1577fa7712a7c9ec8c1614dc"} Jul 06 19:29:22.499616 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:22.499507 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c7f5bdd96-tkv4l" event={"ID":"2c8d6b42-b114-44e7-bc29-4f1552633e17","Type":"ContainerStarted","Data":"cf8c5d6627d950ff50ba295d027ef11995eb82897ab17aeabdb840db00f17131"} Jul 06 19:29:26.071734 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:26.071665 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5f5769b66b-86k92" Jul 06 19:29:26.071734 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:26.071722 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5f5769b66b-86k92" Jul 06 19:29:28.483523 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:28.483492 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-272x4" Jul 06 19:29:28.510086 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:28.510042 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-c7f5bdd96-tkv4l" podStartSLOduration=7.510027343 podStartE2EDuration="7.510027343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:29:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:29:22.523730429 +0000 UTC m=+171.343819168" watchObservedRunningTime="2026-07-06 19:29:28.510027343 +0000 UTC m=+177.330116055" Jul 06 19:29:31.815405 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:31.815373 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c7f5bdd96-tkv4l" Jul 06 19:29:31.815829 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:31.815423 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c7f5bdd96-tkv4l" Jul 06 19:29:31.820301 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:31.820281 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c7f5bdd96-tkv4l" Jul 06 19:29:32.544420 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:32.544394 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c7f5bdd96-tkv4l" Jul 06 19:29:32.617512 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:32.617478 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d74c98c7c-kzskh"] Jul 06 19:29:41.577841 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:41.577799 2575 generic.go:356] "Generic (PLEG): container finished" podID="e3efc9cc-6d76-4cde-a12a-c0f5022e1abf" containerID="fe09003cef97f6a6d47f53f334ca46c09cd53d800d6dc55bbceef00b402aebdd" exitCode=0 Jul 06 19:29:41.578381 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:41.577881 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-5wmbj" event={"ID":"e3efc9cc-6d76-4cde-a12a-c0f5022e1abf","Type":"ContainerDied","Data":"fe09003cef97f6a6d47f53f334ca46c09cd53d800d6dc55bbceef00b402aebdd"} Jul 06 19:29:41.578448 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:41.578422 2575 scope.go:117] "RemoveContainer" containerID="fe09003cef97f6a6d47f53f334ca46c09cd53d800d6dc55bbceef00b402aebdd" Jul 06 19:29:42.587726 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:42.585667 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-5wmbj" event={"ID":"e3efc9cc-6d76-4cde-a12a-c0f5022e1abf","Type":"ContainerStarted","Data":"1231a9bc267c0080b5660c852ed5fcfc687e0bb2359e60d5ff47d316d14a9f27"} Jul 06 19:29:42.617209 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:42.617148 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-7bf496c95-7q6sp" podUID="6b07f9de-449f-4154-b0ee-2d06d6a93313" containerName="console" containerID="cri-o://83ab831541e3f744ea44f1ec0ab8505e5f7cc1ddae7740fe393266a8064a3f36" gracePeriod=15 Jul 06 19:29:42.897000 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:42.896975 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bf496c95-7q6sp" Jul 06 19:29:43.019766 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:43.019733 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b07f9de-449f-4154-b0ee-2d06d6a93313-service-ca\") pod \"6b07f9de-449f-4154-b0ee-2d06d6a93313\" (UID: \"6b07f9de-449f-4154-b0ee-2d06d6a93313\") " Jul 06 19:29:43.019951 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:43.019794 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b07f9de-449f-4154-b0ee-2d06d6a93313-oauth-serving-cert\") pod \"6b07f9de-449f-4154-b0ee-2d06d6a93313\" (UID: \"6b07f9de-449f-4154-b0ee-2d06d6a93313\") " Jul 06 19:29:43.019951 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:43.019865 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b07f9de-449f-4154-b0ee-2d06d6a93313-console-oauth-config\") pod \"6b07f9de-449f-4154-b0ee-2d06d6a93313\" (UID: \"6b07f9de-449f-4154-b0ee-2d06d6a93313\") " Jul 06 19:29:43.019951 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:43.019888 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b07f9de-449f-4154-b0ee-2d06d6a93313-console-config\") pod \"6b07f9de-449f-4154-b0ee-2d06d6a93313\" (UID: \"6b07f9de-449f-4154-b0ee-2d06d6a93313\") " Jul 06 19:29:43.020119 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:43.020001 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b07f9de-449f-4154-b0ee-2d06d6a93313-console-serving-cert\") pod \"6b07f9de-449f-4154-b0ee-2d06d6a93313\" (UID: \"6b07f9de-449f-4154-b0ee-2d06d6a93313\") " Jul 06 19:29:43.020119 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:43.020045 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qpdl\" (UniqueName: \"kubernetes.io/projected/6b07f9de-449f-4154-b0ee-2d06d6a93313-kube-api-access-8qpdl\") pod \"6b07f9de-449f-4154-b0ee-2d06d6a93313\" (UID: \"6b07f9de-449f-4154-b0ee-2d06d6a93313\") " Jul 06 19:29:43.020262 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:43.020199 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b07f9de-449f-4154-b0ee-2d06d6a93313-service-ca" (OuterVolumeSpecName: "service-ca") pod "6b07f9de-449f-4154-b0ee-2d06d6a93313" (UID: "6b07f9de-449f-4154-b0ee-2d06d6a93313"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:29:43.020320 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:43.020256 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b07f9de-449f-4154-b0ee-2d06d6a93313-console-config" (OuterVolumeSpecName: "console-config") pod "6b07f9de-449f-4154-b0ee-2d06d6a93313" (UID: "6b07f9de-449f-4154-b0ee-2d06d6a93313"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:29:43.020320 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:43.020264 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b07f9de-449f-4154-b0ee-2d06d6a93313-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6b07f9de-449f-4154-b0ee-2d06d6a93313" (UID: "6b07f9de-449f-4154-b0ee-2d06d6a93313"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:29:43.020432 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:43.020413 2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b07f9de-449f-4154-b0ee-2d06d6a93313-oauth-serving-cert\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:29:43.020480 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:43.020438 2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b07f9de-449f-4154-b0ee-2d06d6a93313-console-config\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:29:43.020480 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:43.020452 2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b07f9de-449f-4154-b0ee-2d06d6a93313-service-ca\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:29:43.022098 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:43.022075 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b07f9de-449f-4154-b0ee-2d06d6a93313-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6b07f9de-449f-4154-b0ee-2d06d6a93313" (UID: "6b07f9de-449f-4154-b0ee-2d06d6a93313"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:29:43.022274 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:43.022113 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b07f9de-449f-4154-b0ee-2d06d6a93313-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6b07f9de-449f-4154-b0ee-2d06d6a93313" (UID: "6b07f9de-449f-4154-b0ee-2d06d6a93313"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:29:43.022274 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:43.022239 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b07f9de-449f-4154-b0ee-2d06d6a93313-kube-api-access-8qpdl" (OuterVolumeSpecName: "kube-api-access-8qpdl") pod "6b07f9de-449f-4154-b0ee-2d06d6a93313" (UID: "6b07f9de-449f-4154-b0ee-2d06d6a93313"). InnerVolumeSpecName "kube-api-access-8qpdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:29:43.121243 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:43.121159 2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b07f9de-449f-4154-b0ee-2d06d6a93313-console-oauth-config\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:29:43.121243 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:43.121205 2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b07f9de-449f-4154-b0ee-2d06d6a93313-console-serving-cert\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:29:43.121243 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:43.121215 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qpdl\" (UniqueName: \"kubernetes.io/projected/6b07f9de-449f-4154-b0ee-2d06d6a93313-kube-api-access-8qpdl\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:29:43.595626 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:43.595592 2575 generic.go:356] "Generic (PLEG): container finished" podID="6b07f9de-449f-4154-b0ee-2d06d6a93313" containerID="83ab831541e3f744ea44f1ec0ab8505e5f7cc1ddae7740fe393266a8064a3f36" exitCode=0 Jul 06 19:29:43.596032 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:43.595638 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bf496c95-7q6sp" Jul 06 19:29:43.596032 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:43.595656 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bf496c95-7q6sp" event={"ID":"6b07f9de-449f-4154-b0ee-2d06d6a93313","Type":"ContainerDied","Data":"83ab831541e3f744ea44f1ec0ab8505e5f7cc1ddae7740fe393266a8064a3f36"} Jul 06 19:29:43.596032 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:43.595732 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bf496c95-7q6sp" event={"ID":"6b07f9de-449f-4154-b0ee-2d06d6a93313","Type":"ContainerDied","Data":"dfae2d9f8c94d9323585a1b4139f2176cf1ad505f711a5a1c92e39672229c2cd"} Jul 06 19:29:43.596032 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:43.595758 2575 scope.go:117] "RemoveContainer" containerID="83ab831541e3f744ea44f1ec0ab8505e5f7cc1ddae7740fe393266a8064a3f36" Jul 06 19:29:43.604850 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:43.604833 2575 scope.go:117] "RemoveContainer" containerID="83ab831541e3f744ea44f1ec0ab8505e5f7cc1ddae7740fe393266a8064a3f36" Jul 06 19:29:43.605120 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:29:43.605101 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83ab831541e3f744ea44f1ec0ab8505e5f7cc1ddae7740fe393266a8064a3f36\": container with ID starting with 83ab831541e3f744ea44f1ec0ab8505e5f7cc1ddae7740fe393266a8064a3f36 not found: ID does not exist" containerID="83ab831541e3f744ea44f1ec0ab8505e5f7cc1ddae7740fe393266a8064a3f36" Jul 06 19:29:43.605168 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:43.605127 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83ab831541e3f744ea44f1ec0ab8505e5f7cc1ddae7740fe393266a8064a3f36"} err="failed to get container status \"83ab831541e3f744ea44f1ec0ab8505e5f7cc1ddae7740fe393266a8064a3f36\": rpc error: code = NotFound desc = could not find container \"83ab831541e3f744ea44f1ec0ab8505e5f7cc1ddae7740fe393266a8064a3f36\": container with ID starting with 83ab831541e3f744ea44f1ec0ab8505e5f7cc1ddae7740fe393266a8064a3f36 not found: ID does not exist" Jul 06 19:29:43.633411 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:43.633385 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bf496c95-7q6sp"] Jul 06 19:29:43.635419 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:43.635396 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7bf496c95-7q6sp"] Jul 06 19:29:43.793289 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:43.793259 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b07f9de-449f-4154-b0ee-2d06d6a93313" path="/var/lib/kubelet/pods/6b07f9de-449f-4154-b0ee-2d06d6a93313/volumes" Jul 06 19:29:46.077196 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:46.077152 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5f5769b66b-86k92" Jul 06 19:29:46.080993 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:46.080975 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5f5769b66b-86k92" Jul 06 19:29:57.638229 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:57.638170 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-5d74c98c7c-kzskh" podUID="aa5d74e3-4974-4e1f-bbfd-4ffc679e7780" containerName="console" containerID="cri-o://7fe864f176a976b0a923726acc9f1a9758a329b415a5b941562d69c495619079" gracePeriod=15 Jul 06 19:29:57.921083 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:57.921059 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d74c98c7c-kzskh" Jul 06 19:29:57.936652 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:57.936629 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vs5jj\" (UniqueName: \"kubernetes.io/projected/aa5d74e3-4974-4e1f-bbfd-4ffc679e7780-kube-api-access-vs5jj\") pod \"aa5d74e3-4974-4e1f-bbfd-4ffc679e7780\" (UID: \"aa5d74e3-4974-4e1f-bbfd-4ffc679e7780\") " Jul 06 19:29:57.936748 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:57.936657 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa5d74e3-4974-4e1f-bbfd-4ffc679e7780-trusted-ca-bundle\") pod \"aa5d74e3-4974-4e1f-bbfd-4ffc679e7780\" (UID: \"aa5d74e3-4974-4e1f-bbfd-4ffc679e7780\") " Jul 06 19:29:57.936748 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:57.936688 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa5d74e3-4974-4e1f-bbfd-4ffc679e7780-oauth-serving-cert\") pod \"aa5d74e3-4974-4e1f-bbfd-4ffc679e7780\" (UID: \"aa5d74e3-4974-4e1f-bbfd-4ffc679e7780\") " Jul 06 19:29:57.936748 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:57.936737 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa5d74e3-4974-4e1f-bbfd-4ffc679e7780-console-oauth-config\") pod \"aa5d74e3-4974-4e1f-bbfd-4ffc679e7780\" (UID: \"aa5d74e3-4974-4e1f-bbfd-4ffc679e7780\") " Jul 06 19:29:57.936854 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:57.936772 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa5d74e3-4974-4e1f-bbfd-4ffc679e7780-service-ca\") pod \"aa5d74e3-4974-4e1f-bbfd-4ffc679e7780\" (UID: \"aa5d74e3-4974-4e1f-bbfd-4ffc679e7780\") " Jul 06 19:29:57.936854 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:57.936799 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa5d74e3-4974-4e1f-bbfd-4ffc679e7780-console-serving-cert\") pod \"aa5d74e3-4974-4e1f-bbfd-4ffc679e7780\" (UID: \"aa5d74e3-4974-4e1f-bbfd-4ffc679e7780\") " Jul 06 19:29:57.936854 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:57.936829 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa5d74e3-4974-4e1f-bbfd-4ffc679e7780-console-config\") pod \"aa5d74e3-4974-4e1f-bbfd-4ffc679e7780\" (UID: \"aa5d74e3-4974-4e1f-bbfd-4ffc679e7780\") " Jul 06 19:29:57.937079 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:57.937043 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa5d74e3-4974-4e1f-bbfd-4ffc679e7780-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "aa5d74e3-4974-4e1f-bbfd-4ffc679e7780" (UID: "aa5d74e3-4974-4e1f-bbfd-4ffc679e7780"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:29:57.937155 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:57.937110 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa5d74e3-4974-4e1f-bbfd-4ffc679e7780-service-ca" (OuterVolumeSpecName: "service-ca") pod "aa5d74e3-4974-4e1f-bbfd-4ffc679e7780" (UID: "aa5d74e3-4974-4e1f-bbfd-4ffc679e7780"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:29:57.937155 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:57.937131 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa5d74e3-4974-4e1f-bbfd-4ffc679e7780-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "aa5d74e3-4974-4e1f-bbfd-4ffc679e7780" (UID: "aa5d74e3-4974-4e1f-bbfd-4ffc679e7780"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:29:57.937367 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:57.937346 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa5d74e3-4974-4e1f-bbfd-4ffc679e7780-console-config" (OuterVolumeSpecName: "console-config") pod "aa5d74e3-4974-4e1f-bbfd-4ffc679e7780" (UID: "aa5d74e3-4974-4e1f-bbfd-4ffc679e7780"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:29:57.938737 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:57.938717 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa5d74e3-4974-4e1f-bbfd-4ffc679e7780-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "aa5d74e3-4974-4e1f-bbfd-4ffc679e7780" (UID: "aa5d74e3-4974-4e1f-bbfd-4ffc679e7780"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:29:57.938835 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:57.938812 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa5d74e3-4974-4e1f-bbfd-4ffc679e7780-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "aa5d74e3-4974-4e1f-bbfd-4ffc679e7780" (UID: "aa5d74e3-4974-4e1f-bbfd-4ffc679e7780"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:29:57.938876 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:57.938838 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa5d74e3-4974-4e1f-bbfd-4ffc679e7780-kube-api-access-vs5jj" (OuterVolumeSpecName: "kube-api-access-vs5jj") pod "aa5d74e3-4974-4e1f-bbfd-4ffc679e7780" (UID: "aa5d74e3-4974-4e1f-bbfd-4ffc679e7780"). InnerVolumeSpecName "kube-api-access-vs5jj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:29:58.037981 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:58.037960 2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa5d74e3-4974-4e1f-bbfd-4ffc679e7780-oauth-serving-cert\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:29:58.037981 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:58.037981 2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa5d74e3-4974-4e1f-bbfd-4ffc679e7780-console-oauth-config\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:29:58.038115 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:58.037991 2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa5d74e3-4974-4e1f-bbfd-4ffc679e7780-service-ca\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:29:58.038115 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:58.037999 2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa5d74e3-4974-4e1f-bbfd-4ffc679e7780-console-serving-cert\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:29:58.038115 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:58.038008 2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa5d74e3-4974-4e1f-bbfd-4ffc679e7780-console-config\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:29:58.038115 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:58.038016 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs5jj\" (UniqueName: \"kubernetes.io/projected/aa5d74e3-4974-4e1f-bbfd-4ffc679e7780-kube-api-access-vs5jj\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:29:58.038115 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:58.038024 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa5d74e3-4974-4e1f-bbfd-4ffc679e7780-trusted-ca-bundle\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:29:58.661630 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:58.661596 2575 generic.go:356] "Generic (PLEG): container finished" podID="aa5d74e3-4974-4e1f-bbfd-4ffc679e7780" containerID="7fe864f176a976b0a923726acc9f1a9758a329b415a5b941562d69c495619079" exitCode=0 Jul 06 19:29:58.662002 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:58.661644 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d74c98c7c-kzskh" Jul 06 19:29:58.662002 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:58.661686 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d74c98c7c-kzskh" event={"ID":"aa5d74e3-4974-4e1f-bbfd-4ffc679e7780","Type":"ContainerDied","Data":"7fe864f176a976b0a923726acc9f1a9758a329b415a5b941562d69c495619079"} Jul 06 19:29:58.662002 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:58.661736 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d74c98c7c-kzskh" event={"ID":"aa5d74e3-4974-4e1f-bbfd-4ffc679e7780","Type":"ContainerDied","Data":"209bbfd4239b964f5710befab51db12284a66e370cbc90bc7f2386e98169cab8"} Jul 06 19:29:58.662002 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:58.661759 2575 scope.go:117] "RemoveContainer" containerID="7fe864f176a976b0a923726acc9f1a9758a329b415a5b941562d69c495619079" Jul 06 19:29:58.670197 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:58.670158 2575 scope.go:117] "RemoveContainer" containerID="7fe864f176a976b0a923726acc9f1a9758a329b415a5b941562d69c495619079" Jul 06 19:29:58.670431 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:29:58.670414 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fe864f176a976b0a923726acc9f1a9758a329b415a5b941562d69c495619079\": container with ID starting with 7fe864f176a976b0a923726acc9f1a9758a329b415a5b941562d69c495619079 not found: ID does not exist" containerID="7fe864f176a976b0a923726acc9f1a9758a329b415a5b941562d69c495619079" Jul 06 19:29:58.670483 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:58.670438 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fe864f176a976b0a923726acc9f1a9758a329b415a5b941562d69c495619079"} err="failed to get container status \"7fe864f176a976b0a923726acc9f1a9758a329b415a5b941562d69c495619079\": rpc error: code = NotFound desc = could not find container \"7fe864f176a976b0a923726acc9f1a9758a329b415a5b941562d69c495619079\": container with ID starting with 7fe864f176a976b0a923726acc9f1a9758a329b415a5b941562d69c495619079 not found: ID does not exist" Jul 06 19:29:58.684649 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:58.684630 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d74c98c7c-kzskh"] Jul 06 19:29:58.691545 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:58.691527 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d74c98c7c-kzskh"] Jul 06 19:29:59.796914 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:29:59.794489 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa5d74e3-4974-4e1f-bbfd-4ffc679e7780" path="/var/lib/kubelet/pods/aa5d74e3-4974-4e1f-bbfd-4ffc679e7780/volumes" Jul 06 19:30:23.330363 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.330334 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 19:30:23.330816 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.330623 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa5d74e3-4974-4e1f-bbfd-4ffc679e7780" containerName="console" Jul 06 19:30:23.330816 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.330633 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa5d74e3-4974-4e1f-bbfd-4ffc679e7780" containerName="console" Jul 06 19:30:23.330816 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.330649 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b07f9de-449f-4154-b0ee-2d06d6a93313" containerName="console" Jul 06 19:30:23.330816 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.330655 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b07f9de-449f-4154-b0ee-2d06d6a93313" containerName="console" Jul 06 19:30:23.330816 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.330709 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b07f9de-449f-4154-b0ee-2d06d6a93313" containerName="console" Jul 06 19:30:23.330816 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.330721 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa5d74e3-4974-4e1f-bbfd-4ffc679e7780" containerName="console" Jul 06 19:30:23.333042 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.333026 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:30:23.337398 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.337375 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 06 19:30:23.337510 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.337375 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 06 19:30:23.338237 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.338218 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-thr8m\"" Jul 06 19:30:23.340162 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.340142 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 06 19:30:23.340419 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.340160 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 06 19:30:23.340419 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.340295 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 06 19:30:23.341958 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.341939 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 06 19:30:23.342041 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.341960 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 06 19:30:23.343379 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.343360 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 06 19:30:23.363064 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.363044 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 19:30:23.365114 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.365087 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 06 19:30:23.434872 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.434852 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/24edef41-b63d-4fe8-9bd9-c3f28160cde8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"24edef41-b63d-4fe8-9bd9-c3f28160cde8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:30:23.434982 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.434883 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24edef41-b63d-4fe8-9bd9-c3f28160cde8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"24edef41-b63d-4fe8-9bd9-c3f28160cde8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:30:23.434982 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.434902 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/24edef41-b63d-4fe8-9bd9-c3f28160cde8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"24edef41-b63d-4fe8-9bd9-c3f28160cde8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:30:23.434982 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.434923 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24edef41-b63d-4fe8-9bd9-c3f28160cde8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"24edef41-b63d-4fe8-9bd9-c3f28160cde8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:30:23.435140 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.434979 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/24edef41-b63d-4fe8-9bd9-c3f28160cde8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"24edef41-b63d-4fe8-9bd9-c3f28160cde8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:30:23.435140 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.435025 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/24edef41-b63d-4fe8-9bd9-c3f28160cde8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"24edef41-b63d-4fe8-9bd9-c3f28160cde8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:30:23.435140 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.435044 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/24edef41-b63d-4fe8-9bd9-c3f28160cde8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"24edef41-b63d-4fe8-9bd9-c3f28160cde8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:30:23.435140 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.435058 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/24edef41-b63d-4fe8-9bd9-c3f28160cde8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"24edef41-b63d-4fe8-9bd9-c3f28160cde8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:30:23.435140 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.435078 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/24edef41-b63d-4fe8-9bd9-c3f28160cde8-web-config\") pod \"alertmanager-main-0\" (UID: \"24edef41-b63d-4fe8-9bd9-c3f28160cde8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:30:23.435416 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.435133 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/24edef41-b63d-4fe8-9bd9-c3f28160cde8-config-out\") pod \"alertmanager-main-0\" (UID: \"24edef41-b63d-4fe8-9bd9-c3f28160cde8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:30:23.435416 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.435169 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx4zr\" (UniqueName: \"kubernetes.io/projected/24edef41-b63d-4fe8-9bd9-c3f28160cde8-kube-api-access-gx4zr\") pod \"alertmanager-main-0\" (UID: \"24edef41-b63d-4fe8-9bd9-c3f28160cde8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:30:23.435416 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.435235 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/24edef41-b63d-4fe8-9bd9-c3f28160cde8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"24edef41-b63d-4fe8-9bd9-c3f28160cde8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:30:23.435416 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.435293 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/24edef41-b63d-4fe8-9bd9-c3f28160cde8-config-volume\") pod \"alertmanager-main-0\" (UID: \"24edef41-b63d-4fe8-9bd9-c3f28160cde8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:30:23.536292 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.536266 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24edef41-b63d-4fe8-9bd9-c3f28160cde8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"24edef41-b63d-4fe8-9bd9-c3f28160cde8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:30:23.536401 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.536299 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/24edef41-b63d-4fe8-9bd9-c3f28160cde8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"24edef41-b63d-4fe8-9bd9-c3f28160cde8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:30:23.536401 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.536331 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24edef41-b63d-4fe8-9bd9-c3f28160cde8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"24edef41-b63d-4fe8-9bd9-c3f28160cde8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:30:23.536401 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.536361 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/24edef41-b63d-4fe8-9bd9-c3f28160cde8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"24edef41-b63d-4fe8-9bd9-c3f28160cde8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:30:23.536557 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.536400 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/24edef41-b63d-4fe8-9bd9-c3f28160cde8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"24edef41-b63d-4fe8-9bd9-c3f28160cde8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:30:23.536557 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.536426 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/24edef41-b63d-4fe8-9bd9-c3f28160cde8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"24edef41-b63d-4fe8-9bd9-c3f28160cde8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:30:23.536557 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.536453 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/24edef41-b63d-4fe8-9bd9-c3f28160cde8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"24edef41-b63d-4fe8-9bd9-c3f28160cde8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:30:23.536557 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.536499 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/24edef41-b63d-4fe8-9bd9-c3f28160cde8-web-config\") pod \"alertmanager-main-0\" (UID: \"24edef41-b63d-4fe8-9bd9-c3f28160cde8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:30:23.536557 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.536529 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/24edef41-b63d-4fe8-9bd9-c3f28160cde8-config-out\") pod \"alertmanager-main-0\" (UID: \"24edef41-b63d-4fe8-9bd9-c3f28160cde8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:30:23.536557 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.536553 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx4zr\" (UniqueName: \"kubernetes.io/projected/24edef41-b63d-4fe8-9bd9-c3f28160cde8-kube-api-access-gx4zr\") pod \"alertmanager-main-0\" (UID: \"24edef41-b63d-4fe8-9bd9-c3f28160cde8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:30:23.536842 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.536581 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/24edef41-b63d-4fe8-9bd9-c3f28160cde8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"24edef41-b63d-4fe8-9bd9-c3f28160cde8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:30:23.536842 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.536623 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/24edef41-b63d-4fe8-9bd9-c3f28160cde8-config-volume\") pod \"alertmanager-main-0\" (UID: \"24edef41-b63d-4fe8-9bd9-c3f28160cde8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:30:23.536842 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.536695 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/24edef41-b63d-4fe8-9bd9-c3f28160cde8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"24edef41-b63d-4fe8-9bd9-c3f28160cde8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:30:23.536991 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.536848 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/24edef41-b63d-4fe8-9bd9-c3f28160cde8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"24edef41-b63d-4fe8-9bd9-c3f28160cde8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:30:23.537090 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.537068 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24edef41-b63d-4fe8-9bd9-c3f28160cde8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"24edef41-b63d-4fe8-9bd9-c3f28160cde8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:30:23.537348 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.537323 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24edef41-b63d-4fe8-9bd9-c3f28160cde8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"24edef41-b63d-4fe8-9bd9-c3f28160cde8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:30:23.539691 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.539648 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/24edef41-b63d-4fe8-9bd9-c3f28160cde8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"24edef41-b63d-4fe8-9bd9-c3f28160cde8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:30:23.539782 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.539740 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/24edef41-b63d-4fe8-9bd9-c3f28160cde8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"24edef41-b63d-4fe8-9bd9-c3f28160cde8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:30:23.539847 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.539795 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/24edef41-b63d-4fe8-9bd9-c3f28160cde8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"24edef41-b63d-4fe8-9bd9-c3f28160cde8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:30:23.540195 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.540156 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/24edef41-b63d-4fe8-9bd9-c3f28160cde8-config-out\") pod \"alertmanager-main-0\" (UID: \"24edef41-b63d-4fe8-9bd9-c3f28160cde8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:30:23.540287 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.540223 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/24edef41-b63d-4fe8-9bd9-c3f28160cde8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"24edef41-b63d-4fe8-9bd9-c3f28160cde8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:30:23.540335 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.540285 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/24edef41-b63d-4fe8-9bd9-c3f28160cde8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"24edef41-b63d-4fe8-9bd9-c3f28160cde8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:30:23.540376 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.540333 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/24edef41-b63d-4fe8-9bd9-c3f28160cde8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"24edef41-b63d-4fe8-9bd9-c3f28160cde8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:30:23.540556 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.540538 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/24edef41-b63d-4fe8-9bd9-c3f28160cde8-web-config\") pod \"alertmanager-main-0\" (UID: \"24edef41-b63d-4fe8-9bd9-c3f28160cde8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:30:23.541560 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.541538 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/24edef41-b63d-4fe8-9bd9-c3f28160cde8-config-volume\") pod \"alertmanager-main-0\" (UID: \"24edef41-b63d-4fe8-9bd9-c3f28160cde8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:30:23.548574 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.548554 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx4zr\" (UniqueName: \"kubernetes.io/projected/24edef41-b63d-4fe8-9bd9-c3f28160cde8-kube-api-access-gx4zr\") pod \"alertmanager-main-0\" (UID: \"24edef41-b63d-4fe8-9bd9-c3f28160cde8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:30:23.644168 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.644113 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:30:23.781790 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:23.781717 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 19:30:23.784434 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:30:23.784403 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24edef41_b63d_4fe8_9bd9_c3f28160cde8.slice/crio-395b52117c1d6a849e8527900f428513e6ac3a089612bed10d2c0ef4ad0fe52c WatchSource:0}: Error finding container 395b52117c1d6a849e8527900f428513e6ac3a089612bed10d2c0ef4ad0fe52c: Status 404 returned error can't find the container with id 395b52117c1d6a849e8527900f428513e6ac3a089612bed10d2c0ef4ad0fe52c Jul 06 19:30:24.767049 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:24.767014 2575 generic.go:356] "Generic (PLEG): container finished" podID="24edef41-b63d-4fe8-9bd9-c3f28160cde8" containerID="cdccadc2b6fde6bf55d9847261cd057f94519014c7660515417554f37bd2426a" exitCode=0 Jul 06 19:30:24.767541 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:24.767060 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"24edef41-b63d-4fe8-9bd9-c3f28160cde8","Type":"ContainerDied","Data":"cdccadc2b6fde6bf55d9847261cd057f94519014c7660515417554f37bd2426a"} Jul 06 19:30:24.767541 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:24.767128 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"24edef41-b63d-4fe8-9bd9-c3f28160cde8","Type":"ContainerStarted","Data":"395b52117c1d6a849e8527900f428513e6ac3a089612bed10d2c0ef4ad0fe52c"} Jul 06 19:30:26.779415 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:26.779379 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"24edef41-b63d-4fe8-9bd9-c3f28160cde8","Type":"ContainerStarted","Data":"69a7c566731e3e5d289b127fa71c437cc89b846936187af302eb8dc384bc3327"} Jul 06 19:30:26.779712 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:26.779454 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"24edef41-b63d-4fe8-9bd9-c3f28160cde8","Type":"ContainerStarted","Data":"cb4c78889fba09c30ab9e1e2c04af07c7936c8e92b92726183c1b565f778677f"} Jul 06 19:30:26.779712 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:26.779495 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"24edef41-b63d-4fe8-9bd9-c3f28160cde8","Type":"ContainerStarted","Data":"edbd60b7047bfa170adfb58457aa09f8b1f51bbb4bcf1053c099107695fd319d"} Jul 06 19:30:26.779712 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:26.779521 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"24edef41-b63d-4fe8-9bd9-c3f28160cde8","Type":"ContainerStarted","Data":"6ec0235cd47c3f1d09134734b29b96f97ae33f8116ce9d239ee6ebba658f2076"} Jul 06 19:30:26.779712 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:26.779550 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"24edef41-b63d-4fe8-9bd9-c3f28160cde8","Type":"ContainerStarted","Data":"42eace9643f80822a085f3a149f3e471960bda458d641571272cf25c87ea911f"} Jul 06 19:30:26.779712 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:26.779581 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"24edef41-b63d-4fe8-9bd9-c3f28160cde8","Type":"ContainerStarted","Data":"81c3fd1df09b6c793f709189d89316dbd8661fb6ddb16381755b5f8414f57f3f"} Jul 06 19:30:26.817250 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:26.815092 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.2658008880000002 podStartE2EDuration="3.815071032s" totalImagesPullingTime="1.53379661s" totalInitContainerRuntime="15.473534ms" isStatefulPod=false podCreationTimestamp="2026-07-06 19:30:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:30:26.810465387 +0000 UTC m=+235.630554136" watchObservedRunningTime="2026-07-06 19:30:26.815071032 +0000 UTC m=+235.635159745" Jul 06 19:30:35.290446 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:35.290414 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-69ff557b65-zcl5k"] Jul 06 19:30:35.293804 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:35.293783 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69ff557b65-zcl5k" Jul 06 19:30:35.306428 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:35.306405 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69ff557b65-zcl5k"] Jul 06 19:30:35.431819 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:35.431786 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/21b51266-ec66-4a9c-aed6-f25fa6b55785-console-config\") pod \"console-69ff557b65-zcl5k\" (UID: \"21b51266-ec66-4a9c-aed6-f25fa6b55785\") " pod="openshift-console/console-69ff557b65-zcl5k" Jul 06 19:30:35.431973 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:35.431842 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/21b51266-ec66-4a9c-aed6-f25fa6b55785-service-ca\") pod \"console-69ff557b65-zcl5k\" (UID: \"21b51266-ec66-4a9c-aed6-f25fa6b55785\") " pod="openshift-console/console-69ff557b65-zcl5k" Jul 06 19:30:35.431973 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:35.431866 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/21b51266-ec66-4a9c-aed6-f25fa6b55785-console-oauth-config\") pod \"console-69ff557b65-zcl5k\" (UID: \"21b51266-ec66-4a9c-aed6-f25fa6b55785\") " pod="openshift-console/console-69ff557b65-zcl5k" Jul 06 19:30:35.431973 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:35.431903 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/21b51266-ec66-4a9c-aed6-f25fa6b55785-oauth-serving-cert\") pod \"console-69ff557b65-zcl5k\" (UID: \"21b51266-ec66-4a9c-aed6-f25fa6b55785\") " pod="openshift-console/console-69ff557b65-zcl5k" Jul 06 19:30:35.432148 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:35.432002 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw87d\" (UniqueName: \"kubernetes.io/projected/21b51266-ec66-4a9c-aed6-f25fa6b55785-kube-api-access-pw87d\") pod \"console-69ff557b65-zcl5k\" (UID: \"21b51266-ec66-4a9c-aed6-f25fa6b55785\") " pod="openshift-console/console-69ff557b65-zcl5k" Jul 06 19:30:35.432148 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:35.432038 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21b51266-ec66-4a9c-aed6-f25fa6b55785-trusted-ca-bundle\") pod \"console-69ff557b65-zcl5k\" (UID: \"21b51266-ec66-4a9c-aed6-f25fa6b55785\") " pod="openshift-console/console-69ff557b65-zcl5k" Jul 06 19:30:35.432148 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:35.432055 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/21b51266-ec66-4a9c-aed6-f25fa6b55785-console-serving-cert\") pod \"console-69ff557b65-zcl5k\" (UID: \"21b51266-ec66-4a9c-aed6-f25fa6b55785\") " pod="openshift-console/console-69ff557b65-zcl5k" Jul 06 19:30:35.533060 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:35.533026 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/21b51266-ec66-4a9c-aed6-f25fa6b55785-console-config\") pod \"console-69ff557b65-zcl5k\" (UID: \"21b51266-ec66-4a9c-aed6-f25fa6b55785\") " pod="openshift-console/console-69ff557b65-zcl5k" Jul 06 19:30:35.533249 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:35.533081 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/21b51266-ec66-4a9c-aed6-f25fa6b55785-service-ca\") pod \"console-69ff557b65-zcl5k\" (UID: \"21b51266-ec66-4a9c-aed6-f25fa6b55785\") " pod="openshift-console/console-69ff557b65-zcl5k" Jul 06 19:30:35.533249 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:35.533109 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/21b51266-ec66-4a9c-aed6-f25fa6b55785-console-oauth-config\") pod \"console-69ff557b65-zcl5k\" (UID: \"21b51266-ec66-4a9c-aed6-f25fa6b55785\") " pod="openshift-console/console-69ff557b65-zcl5k" Jul 06 19:30:35.533249 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:35.533143 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/21b51266-ec66-4a9c-aed6-f25fa6b55785-oauth-serving-cert\") pod \"console-69ff557b65-zcl5k\" (UID: \"21b51266-ec66-4a9c-aed6-f25fa6b55785\") " pod="openshift-console/console-69ff557b65-zcl5k" Jul 06 19:30:35.533249 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:35.533219 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw87d\" (UniqueName: \"kubernetes.io/projected/21b51266-ec66-4a9c-aed6-f25fa6b55785-kube-api-access-pw87d\") pod \"console-69ff557b65-zcl5k\" (UID: \"21b51266-ec66-4a9c-aed6-f25fa6b55785\") " pod="openshift-console/console-69ff557b65-zcl5k" Jul 06 19:30:35.533470 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:35.533254 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21b51266-ec66-4a9c-aed6-f25fa6b55785-trusted-ca-bundle\") pod \"console-69ff557b65-zcl5k\" (UID: \"21b51266-ec66-4a9c-aed6-f25fa6b55785\") " pod="openshift-console/console-69ff557b65-zcl5k" Jul 06 19:30:35.533470 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:35.533280 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/21b51266-ec66-4a9c-aed6-f25fa6b55785-console-serving-cert\") pod \"console-69ff557b65-zcl5k\" (UID: \"21b51266-ec66-4a9c-aed6-f25fa6b55785\") " pod="openshift-console/console-69ff557b65-zcl5k" Jul 06 19:30:35.533841 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:35.533814 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/21b51266-ec66-4a9c-aed6-f25fa6b55785-console-config\") pod \"console-69ff557b65-zcl5k\" (UID: \"21b51266-ec66-4a9c-aed6-f25fa6b55785\") " pod="openshift-console/console-69ff557b65-zcl5k" Jul 06 19:30:35.533841 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:35.533831 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/21b51266-ec66-4a9c-aed6-f25fa6b55785-service-ca\") pod \"console-69ff557b65-zcl5k\" (UID: \"21b51266-ec66-4a9c-aed6-f25fa6b55785\") " pod="openshift-console/console-69ff557b65-zcl5k" Jul 06 19:30:35.534015 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:35.533967 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21b51266-ec66-4a9c-aed6-f25fa6b55785-trusted-ca-bundle\") pod \"console-69ff557b65-zcl5k\" (UID: \"21b51266-ec66-4a9c-aed6-f25fa6b55785\") " pod="openshift-console/console-69ff557b65-zcl5k" Jul 06 19:30:35.534015 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:35.533981 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/21b51266-ec66-4a9c-aed6-f25fa6b55785-oauth-serving-cert\") pod \"console-69ff557b65-zcl5k\" (UID: \"21b51266-ec66-4a9c-aed6-f25fa6b55785\") " pod="openshift-console/console-69ff557b65-zcl5k" Jul 06 19:30:35.535577 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:35.535548 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/21b51266-ec66-4a9c-aed6-f25fa6b55785-console-oauth-config\") pod \"console-69ff557b65-zcl5k\" (UID: \"21b51266-ec66-4a9c-aed6-f25fa6b55785\") " pod="openshift-console/console-69ff557b65-zcl5k" Jul 06 19:30:35.535746 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:35.535730 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/21b51266-ec66-4a9c-aed6-f25fa6b55785-console-serving-cert\") pod \"console-69ff557b65-zcl5k\" (UID: \"21b51266-ec66-4a9c-aed6-f25fa6b55785\") " pod="openshift-console/console-69ff557b65-zcl5k" Jul 06 19:30:35.543312 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:35.543258 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw87d\" (UniqueName: \"kubernetes.io/projected/21b51266-ec66-4a9c-aed6-f25fa6b55785-kube-api-access-pw87d\") pod \"console-69ff557b65-zcl5k\" (UID: \"21b51266-ec66-4a9c-aed6-f25fa6b55785\") " pod="openshift-console/console-69ff557b65-zcl5k" Jul 06 19:30:35.602375 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:35.602347 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69ff557b65-zcl5k" Jul 06 19:30:35.732708 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:35.732687 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69ff557b65-zcl5k"] Jul 06 19:30:35.735014 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:30:35.734983 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21b51266_ec66_4a9c_aed6_f25fa6b55785.slice/crio-af827904148dc7a67e55eb13075dc80df77febee2e9a7d416583a185297efb61 WatchSource:0}: Error finding container af827904148dc7a67e55eb13075dc80df77febee2e9a7d416583a185297efb61: Status 404 returned error can't find the container with id af827904148dc7a67e55eb13075dc80df77febee2e9a7d416583a185297efb61 Jul 06 19:30:35.819819 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:35.819746 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69ff557b65-zcl5k" event={"ID":"21b51266-ec66-4a9c-aed6-f25fa6b55785","Type":"ContainerStarted","Data":"aa48bcb1196af679d074d1f1e6058b85623ef2041415a6350be2017662d84147"} Jul 06 19:30:35.819819 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:35.819799 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69ff557b65-zcl5k" event={"ID":"21b51266-ec66-4a9c-aed6-f25fa6b55785","Type":"ContainerStarted","Data":"af827904148dc7a67e55eb13075dc80df77febee2e9a7d416583a185297efb61"} Jul 06 19:30:35.847400 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:35.847350 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-69ff557b65-zcl5k" podStartSLOduration=0.84733504 podStartE2EDuration="847.33504ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:30:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:30:35.84500299 +0000 UTC m=+244.665091699" watchObservedRunningTime="2026-07-06 19:30:35.84733504 +0000 UTC m=+244.667423819" Jul 06 19:30:45.603372 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:45.603331 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-69ff557b65-zcl5k" Jul 06 19:30:45.603866 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:45.603400 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-69ff557b65-zcl5k" Jul 06 19:30:45.608060 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:45.608034 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-69ff557b65-zcl5k" Jul 06 19:30:45.865439 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:45.865370 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-69ff557b65-zcl5k" Jul 06 19:30:45.927584 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:30:45.927554 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c7f5bdd96-tkv4l"] Jul 06 19:31:10.948948 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:10.948883 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-c7f5bdd96-tkv4l" podUID="2c8d6b42-b114-44e7-bc29-4f1552633e17" containerName="console" containerID="cri-o://ada61909261528e2e1fb178ef40536ffd000264f1577fa7712a7c9ec8c1614dc" gracePeriod=15 Jul 06 19:31:11.196988 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:11.196966 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c7f5bdd96-tkv4l" Jul 06 19:31:11.208438 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:11.208385 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2c8d6b42-b114-44e7-bc29-4f1552633e17-oauth-serving-cert\") pod \"2c8d6b42-b114-44e7-bc29-4f1552633e17\" (UID: \"2c8d6b42-b114-44e7-bc29-4f1552633e17\") " Jul 06 19:31:11.208438 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:11.208416 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxlnl\" (UniqueName: \"kubernetes.io/projected/2c8d6b42-b114-44e7-bc29-4f1552633e17-kube-api-access-xxlnl\") pod \"2c8d6b42-b114-44e7-bc29-4f1552633e17\" (UID: \"2c8d6b42-b114-44e7-bc29-4f1552633e17\") " Jul 06 19:31:11.208438 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:11.208435 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2c8d6b42-b114-44e7-bc29-4f1552633e17-console-config\") pod \"2c8d6b42-b114-44e7-bc29-4f1552633e17\" (UID: \"2c8d6b42-b114-44e7-bc29-4f1552633e17\") " Jul 06 19:31:11.208660 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:11.208463 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2c8d6b42-b114-44e7-bc29-4f1552633e17-console-oauth-config\") pod \"2c8d6b42-b114-44e7-bc29-4f1552633e17\" (UID: \"2c8d6b42-b114-44e7-bc29-4f1552633e17\") " Jul 06 19:31:11.208660 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:11.208582 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2c8d6b42-b114-44e7-bc29-4f1552633e17-console-serving-cert\") pod \"2c8d6b42-b114-44e7-bc29-4f1552633e17\" (UID: \"2c8d6b42-b114-44e7-bc29-4f1552633e17\") " Jul 06 19:31:11.208660 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:11.208644 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c8d6b42-b114-44e7-bc29-4f1552633e17-trusted-ca-bundle\") pod \"2c8d6b42-b114-44e7-bc29-4f1552633e17\" (UID: \"2c8d6b42-b114-44e7-bc29-4f1552633e17\") " Jul 06 19:31:11.208814 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:11.208677 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2c8d6b42-b114-44e7-bc29-4f1552633e17-service-ca\") pod \"2c8d6b42-b114-44e7-bc29-4f1552633e17\" (UID: \"2c8d6b42-b114-44e7-bc29-4f1552633e17\") " Jul 06 19:31:11.209074 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:11.208853 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c8d6b42-b114-44e7-bc29-4f1552633e17-console-config" (OuterVolumeSpecName: "console-config") pod "2c8d6b42-b114-44e7-bc29-4f1552633e17" (UID: "2c8d6b42-b114-44e7-bc29-4f1552633e17"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:31:11.209074 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:11.208860 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c8d6b42-b114-44e7-bc29-4f1552633e17-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2c8d6b42-b114-44e7-bc29-4f1552633e17" (UID: "2c8d6b42-b114-44e7-bc29-4f1552633e17"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:31:11.209074 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:11.209015 2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2c8d6b42-b114-44e7-bc29-4f1552633e17-oauth-serving-cert\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:31:11.209074 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:11.209035 2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2c8d6b42-b114-44e7-bc29-4f1552633e17-console-config\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:31:11.209336 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:11.209168 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c8d6b42-b114-44e7-bc29-4f1552633e17-service-ca" (OuterVolumeSpecName: "service-ca") pod "2c8d6b42-b114-44e7-bc29-4f1552633e17" (UID: "2c8d6b42-b114-44e7-bc29-4f1552633e17"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:31:11.209336 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:11.209239 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c8d6b42-b114-44e7-bc29-4f1552633e17-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2c8d6b42-b114-44e7-bc29-4f1552633e17" (UID: "2c8d6b42-b114-44e7-bc29-4f1552633e17"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:31:11.210489 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:11.210469 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c8d6b42-b114-44e7-bc29-4f1552633e17-kube-api-access-xxlnl" (OuterVolumeSpecName: "kube-api-access-xxlnl") pod "2c8d6b42-b114-44e7-bc29-4f1552633e17" (UID: "2c8d6b42-b114-44e7-bc29-4f1552633e17"). InnerVolumeSpecName "kube-api-access-xxlnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:31:11.211014 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:11.210990 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c8d6b42-b114-44e7-bc29-4f1552633e17-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2c8d6b42-b114-44e7-bc29-4f1552633e17" (UID: "2c8d6b42-b114-44e7-bc29-4f1552633e17"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:31:11.211099 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:11.211020 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c8d6b42-b114-44e7-bc29-4f1552633e17-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2c8d6b42-b114-44e7-bc29-4f1552633e17" (UID: "2c8d6b42-b114-44e7-bc29-4f1552633e17"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:31:11.310060 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:11.310027 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxlnl\" (UniqueName: \"kubernetes.io/projected/2c8d6b42-b114-44e7-bc29-4f1552633e17-kube-api-access-xxlnl\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:31:11.310060 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:11.310059 2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2c8d6b42-b114-44e7-bc29-4f1552633e17-console-oauth-config\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:31:11.310060 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:11.310069 2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2c8d6b42-b114-44e7-bc29-4f1552633e17-console-serving-cert\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:31:11.310252 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:11.310077 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c8d6b42-b114-44e7-bc29-4f1552633e17-trusted-ca-bundle\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:31:11.310252 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:11.310086 2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2c8d6b42-b114-44e7-bc29-4f1552633e17-service-ca\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:31:11.965224 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:11.965194 2575 generic.go:356] "Generic (PLEG): container finished" podID="2c8d6b42-b114-44e7-bc29-4f1552633e17" containerID="ada61909261528e2e1fb178ef40536ffd000264f1577fa7712a7c9ec8c1614dc" exitCode=0 Jul 06 19:31:11.965541 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:11.965243 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c7f5bdd96-tkv4l" Jul 06 19:31:11.965541 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:11.965278 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c7f5bdd96-tkv4l" event={"ID":"2c8d6b42-b114-44e7-bc29-4f1552633e17","Type":"ContainerDied","Data":"ada61909261528e2e1fb178ef40536ffd000264f1577fa7712a7c9ec8c1614dc"} Jul 06 19:31:11.965541 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:11.965323 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c7f5bdd96-tkv4l" event={"ID":"2c8d6b42-b114-44e7-bc29-4f1552633e17","Type":"ContainerDied","Data":"cf8c5d6627d950ff50ba295d027ef11995eb82897ab17aeabdb840db00f17131"} Jul 06 19:31:11.965541 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:11.965344 2575 scope.go:117] "RemoveContainer" containerID="ada61909261528e2e1fb178ef40536ffd000264f1577fa7712a7c9ec8c1614dc" Jul 06 19:31:11.973326 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:11.973311 2575 scope.go:117] "RemoveContainer" containerID="ada61909261528e2e1fb178ef40536ffd000264f1577fa7712a7c9ec8c1614dc" Jul 06 19:31:11.973593 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:31:11.973570 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ada61909261528e2e1fb178ef40536ffd000264f1577fa7712a7c9ec8c1614dc\": container with ID starting with ada61909261528e2e1fb178ef40536ffd000264f1577fa7712a7c9ec8c1614dc not found: ID does not exist" containerID="ada61909261528e2e1fb178ef40536ffd000264f1577fa7712a7c9ec8c1614dc" Jul 06 19:31:11.973635 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:11.973595 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ada61909261528e2e1fb178ef40536ffd000264f1577fa7712a7c9ec8c1614dc"} err="failed to get container status \"ada61909261528e2e1fb178ef40536ffd000264f1577fa7712a7c9ec8c1614dc\": rpc error: code = NotFound desc = could not find container \"ada61909261528e2e1fb178ef40536ffd000264f1577fa7712a7c9ec8c1614dc\": container with ID starting with ada61909261528e2e1fb178ef40536ffd000264f1577fa7712a7c9ec8c1614dc not found: ID does not exist" Jul 06 19:31:14.660104 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:14.660073 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c7f5bdd96-tkv4l"] Jul 06 19:31:14.683045 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:14.683018 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c7f5bdd96-tkv4l"] Jul 06 19:31:15.794770 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:15.794737 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c8d6b42-b114-44e7-bc29-4f1552633e17" path="/var/lib/kubelet/pods/2c8d6b42-b114-44e7-bc29-4f1552633e17/volumes" Jul 06 19:31:16.947704 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:16.947675 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pxsmk"] Jul 06 19:31:16.948078 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:16.947999 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c8d6b42-b114-44e7-bc29-4f1552633e17" containerName="console" Jul 06 19:31:16.948078 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:16.948009 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c8d6b42-b114-44e7-bc29-4f1552633e17" containerName="console" Jul 06 19:31:16.948078 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:16.948074 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c8d6b42-b114-44e7-bc29-4f1552633e17" containerName="console" Jul 06 19:31:16.952550 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:16.952534 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pxsmk" Jul 06 19:31:16.955539 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:16.955514 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 06 19:31:16.956601 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:16.956586 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 06 19:31:16.957064 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:16.957040 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-c7vvn\"" Jul 06 19:31:16.965671 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:16.965646 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pxsmk"] Jul 06 19:31:17.054147 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:17.054103 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z6hp\" (UniqueName: \"kubernetes.io/projected/cfab206b-40d3-41e2-840d-38038eb8af27-kube-api-access-2z6hp\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pxsmk\" (UID: \"cfab206b-40d3-41e2-840d-38038eb8af27\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pxsmk" Jul 06 19:31:17.054372 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:17.054219 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cfab206b-40d3-41e2-840d-38038eb8af27-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pxsmk\" (UID: \"cfab206b-40d3-41e2-840d-38038eb8af27\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pxsmk" Jul 06 19:31:17.054372 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:17.054275 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cfab206b-40d3-41e2-840d-38038eb8af27-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pxsmk\" (UID: \"cfab206b-40d3-41e2-840d-38038eb8af27\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pxsmk" Jul 06 19:31:17.154709 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:17.154676 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z6hp\" (UniqueName: \"kubernetes.io/projected/cfab206b-40d3-41e2-840d-38038eb8af27-kube-api-access-2z6hp\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pxsmk\" (UID: \"cfab206b-40d3-41e2-840d-38038eb8af27\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pxsmk" Jul 06 19:31:17.154881 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:17.154739 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cfab206b-40d3-41e2-840d-38038eb8af27-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pxsmk\" (UID: \"cfab206b-40d3-41e2-840d-38038eb8af27\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pxsmk" Jul 06 19:31:17.154881 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:17.154777 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cfab206b-40d3-41e2-840d-38038eb8af27-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pxsmk\" (UID: \"cfab206b-40d3-41e2-840d-38038eb8af27\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pxsmk" Jul 06 19:31:17.155122 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:17.155105 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cfab206b-40d3-41e2-840d-38038eb8af27-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pxsmk\" (UID: \"cfab206b-40d3-41e2-840d-38038eb8af27\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pxsmk" Jul 06 19:31:17.155223 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:17.155202 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cfab206b-40d3-41e2-840d-38038eb8af27-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pxsmk\" (UID: \"cfab206b-40d3-41e2-840d-38038eb8af27\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pxsmk" Jul 06 19:31:17.168425 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:17.168395 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z6hp\" (UniqueName: \"kubernetes.io/projected/cfab206b-40d3-41e2-840d-38038eb8af27-kube-api-access-2z6hp\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pxsmk\" (UID: \"cfab206b-40d3-41e2-840d-38038eb8af27\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pxsmk" Jul 06 19:31:17.267037 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:17.266940 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pxsmk" Jul 06 19:31:17.410248 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:17.410223 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pxsmk"] Jul 06 19:31:17.412876 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:31:17.412839 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfab206b_40d3_41e2_840d_38038eb8af27.slice/crio-8467646fe32f33bf72dff1f8bb96cd337fdc9058d04e015ddec3edf41dfab024 WatchSource:0}: Error finding container 8467646fe32f33bf72dff1f8bb96cd337fdc9058d04e015ddec3edf41dfab024: Status 404 returned error can't find the container with id 8467646fe32f33bf72dff1f8bb96cd337fdc9058d04e015ddec3edf41dfab024 Jul 06 19:31:17.990343 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:17.990315 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pxsmk" event={"ID":"cfab206b-40d3-41e2-840d-38038eb8af27","Type":"ContainerStarted","Data":"8467646fe32f33bf72dff1f8bb96cd337fdc9058d04e015ddec3edf41dfab024"} Jul 06 19:31:26.028657 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:26.028582 2575 generic.go:356] "Generic (PLEG): container finished" podID="cfab206b-40d3-41e2-840d-38038eb8af27" containerID="93215abc416ba06cc87440020f856a865ba70de9f1c414bbf5007035b45bca62" exitCode=0 Jul 06 19:31:26.029042 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:26.028665 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pxsmk" event={"ID":"cfab206b-40d3-41e2-840d-38038eb8af27","Type":"ContainerDied","Data":"93215abc416ba06cc87440020f856a865ba70de9f1c414bbf5007035b45bca62"} Jul 06 19:31:28.045165 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:28.045135 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pxsmk" event={"ID":"cfab206b-40d3-41e2-840d-38038eb8af27","Type":"ContainerStarted","Data":"86eea6c363f1f6979d2eeb66253a26fc7aaae18c390a7413b1e4c1d7dbb84fb8"} Jul 06 19:31:29.051381 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:29.051351 2575 generic.go:356] "Generic (PLEG): container finished" podID="cfab206b-40d3-41e2-840d-38038eb8af27" containerID="86eea6c363f1f6979d2eeb66253a26fc7aaae18c390a7413b1e4c1d7dbb84fb8" exitCode=0 Jul 06 19:31:29.051739 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:29.051419 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pxsmk" event={"ID":"cfab206b-40d3-41e2-840d-38038eb8af27","Type":"ContainerDied","Data":"86eea6c363f1f6979d2eeb66253a26fc7aaae18c390a7413b1e4c1d7dbb84fb8"} Jul 06 19:31:31.618578 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:31.618551 2575 kubelet.go:1648] "Image garbage collection succeeded" Jul 06 19:31:37.090722 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:37.090691 2575 generic.go:356] "Generic (PLEG): container finished" podID="cfab206b-40d3-41e2-840d-38038eb8af27" containerID="8de8e231c78f2ee70d1db472f209b4f321499aaf5731323f8924940573b5f81e" exitCode=0 Jul 06 19:31:37.091092 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:37.090779 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pxsmk" event={"ID":"cfab206b-40d3-41e2-840d-38038eb8af27","Type":"ContainerDied","Data":"8de8e231c78f2ee70d1db472f209b4f321499aaf5731323f8924940573b5f81e"} Jul 06 19:31:38.259926 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:38.259902 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pxsmk" Jul 06 19:31:38.335770 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:38.335743 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cfab206b-40d3-41e2-840d-38038eb8af27-bundle\") pod \"cfab206b-40d3-41e2-840d-38038eb8af27\" (UID: \"cfab206b-40d3-41e2-840d-38038eb8af27\") " Jul 06 19:31:38.335901 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:38.335817 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2z6hp\" (UniqueName: \"kubernetes.io/projected/cfab206b-40d3-41e2-840d-38038eb8af27-kube-api-access-2z6hp\") pod \"cfab206b-40d3-41e2-840d-38038eb8af27\" (UID: \"cfab206b-40d3-41e2-840d-38038eb8af27\") " Jul 06 19:31:38.335901 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:38.335875 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cfab206b-40d3-41e2-840d-38038eb8af27-util\") pod \"cfab206b-40d3-41e2-840d-38038eb8af27\" (UID: \"cfab206b-40d3-41e2-840d-38038eb8af27\") " Jul 06 19:31:38.336478 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:38.336454 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfab206b-40d3-41e2-840d-38038eb8af27-bundle" (OuterVolumeSpecName: "bundle") pod "cfab206b-40d3-41e2-840d-38038eb8af27" (UID: "cfab206b-40d3-41e2-840d-38038eb8af27"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:31:38.337994 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:38.337957 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfab206b-40d3-41e2-840d-38038eb8af27-kube-api-access-2z6hp" (OuterVolumeSpecName: "kube-api-access-2z6hp") pod "cfab206b-40d3-41e2-840d-38038eb8af27" (UID: "cfab206b-40d3-41e2-840d-38038eb8af27"). InnerVolumeSpecName "kube-api-access-2z6hp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:31:38.340655 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:38.340637 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfab206b-40d3-41e2-840d-38038eb8af27-util" (OuterVolumeSpecName: "util") pod "cfab206b-40d3-41e2-840d-38038eb8af27" (UID: "cfab206b-40d3-41e2-840d-38038eb8af27"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:31:38.436987 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:38.436935 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cfab206b-40d3-41e2-840d-38038eb8af27-bundle\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:31:38.436987 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:38.436955 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z6hp\" (UniqueName: \"kubernetes.io/projected/cfab206b-40d3-41e2-840d-38038eb8af27-kube-api-access-2z6hp\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:31:38.436987 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:38.436965 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cfab206b-40d3-41e2-840d-38038eb8af27-util\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:31:39.101404 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:39.101372 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pxsmk" event={"ID":"cfab206b-40d3-41e2-840d-38038eb8af27","Type":"ContainerDied","Data":"8467646fe32f33bf72dff1f8bb96cd337fdc9058d04e015ddec3edf41dfab024"} Jul 06 19:31:39.101564 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:39.101418 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8467646fe32f33bf72dff1f8bb96cd337fdc9058d04e015ddec3edf41dfab024" Jul 06 19:31:39.101564 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:39.101383 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pxsmk" Jul 06 19:31:45.372297 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:45.372254 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lwfnb"] Jul 06 19:31:45.372904 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:45.372758 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfab206b-40d3-41e2-840d-38038eb8af27" containerName="util" Jul 06 19:31:45.372904 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:45.372780 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfab206b-40d3-41e2-840d-38038eb8af27" containerName="util" Jul 06 19:31:45.372904 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:45.372795 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfab206b-40d3-41e2-840d-38038eb8af27" containerName="extract" Jul 06 19:31:45.372904 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:45.372804 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfab206b-40d3-41e2-840d-38038eb8af27" containerName="extract" Jul 06 19:31:45.372904 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:45.372831 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfab206b-40d3-41e2-840d-38038eb8af27" containerName="pull" Jul 06 19:31:45.372904 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:45.372839 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfab206b-40d3-41e2-840d-38038eb8af27" containerName="pull" Jul 06 19:31:45.373333 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:45.372938 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfab206b-40d3-41e2-840d-38038eb8af27" containerName="extract" Jul 06 19:31:45.375358 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:45.375333 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lwfnb" Jul 06 19:31:45.378448 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:45.378428 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-q654z\"" Jul 06 19:31:45.378704 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:45.378687 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:31:45.378782 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:45.378728 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 06 19:31:45.391232 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:45.391202 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lwfnb"] Jul 06 19:31:45.488399 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:45.488373 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88c0a4ea-0097-434c-b463-d93a43433c63-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-lwfnb\" (UID: \"88c0a4ea-0097-434c-b463-d93a43433c63\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lwfnb" Jul 06 19:31:45.488515 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:45.488415 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5pzx\" (UniqueName: \"kubernetes.io/projected/88c0a4ea-0097-434c-b463-d93a43433c63-kube-api-access-x5pzx\") pod \"cert-manager-operator-controller-manager-55c444bf86-lwfnb\" (UID: \"88c0a4ea-0097-434c-b463-d93a43433c63\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lwfnb" Jul 06 19:31:45.588987 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:45.588958 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88c0a4ea-0097-434c-b463-d93a43433c63-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-lwfnb\" (UID: \"88c0a4ea-0097-434c-b463-d93a43433c63\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lwfnb" Jul 06 19:31:45.589105 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:45.589007 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5pzx\" (UniqueName: \"kubernetes.io/projected/88c0a4ea-0097-434c-b463-d93a43433c63-kube-api-access-x5pzx\") pod \"cert-manager-operator-controller-manager-55c444bf86-lwfnb\" (UID: \"88c0a4ea-0097-434c-b463-d93a43433c63\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lwfnb" Jul 06 19:31:45.589394 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:45.589375 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88c0a4ea-0097-434c-b463-d93a43433c63-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-lwfnb\" (UID: \"88c0a4ea-0097-434c-b463-d93a43433c63\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lwfnb" Jul 06 19:31:45.599059 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:45.599041 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5pzx\" (UniqueName: \"kubernetes.io/projected/88c0a4ea-0097-434c-b463-d93a43433c63-kube-api-access-x5pzx\") pod \"cert-manager-operator-controller-manager-55c444bf86-lwfnb\" (UID: \"88c0a4ea-0097-434c-b463-d93a43433c63\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lwfnb" Jul 06 19:31:45.684199 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:45.684129 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lwfnb" Jul 06 19:31:45.816290 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:45.816268 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lwfnb"] Jul 06 19:31:45.819056 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:31:45.819031 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88c0a4ea_0097_434c_b463_d93a43433c63.slice/crio-f2735170f21e4b949663f0533d365d12dad5d725c402bf4872ba57f36abecd05 WatchSource:0}: Error finding container f2735170f21e4b949663f0533d365d12dad5d725c402bf4872ba57f36abecd05: Status 404 returned error can't find the container with id f2735170f21e4b949663f0533d365d12dad5d725c402bf4872ba57f36abecd05 Jul 06 19:31:45.822643 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:45.822629 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:31:46.133108 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:46.133073 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lwfnb" event={"ID":"88c0a4ea-0097-434c-b463-d93a43433c63","Type":"ContainerStarted","Data":"f2735170f21e4b949663f0533d365d12dad5d725c402bf4872ba57f36abecd05"} Jul 06 19:31:48.143586 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:48.143552 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lwfnb" event={"ID":"88c0a4ea-0097-434c-b463-d93a43433c63","Type":"ContainerStarted","Data":"fac769a3ab72677e897908d441e0156c9c61099e17f303e6ac6748ee821353f6"} Jul 06 19:31:48.169570 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:48.169516 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lwfnb" podStartSLOduration=1.292978772 podStartE2EDuration="3.169502369s" totalImagesPullingTime="1.876523597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:31:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:31:48.166872219 +0000 UTC m=+316.986960932" watchObservedRunningTime="2026-07-06 19:31:48.169502369 +0000 UTC m=+316.989591080" Jul 06 19:31:48.970099 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:48.970068 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcpbd9"] Jul 06 19:31:48.972762 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:48.972743 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcpbd9" Jul 06 19:31:48.975871 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:48.975850 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-c7vvn\"" Jul 06 19:31:48.976965 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:48.976947 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 06 19:31:48.977123 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:48.977106 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 06 19:31:49.014285 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:49.014265 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcpbd9"] Jul 06 19:31:49.119015 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:49.118984 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfggf\" (UniqueName: \"kubernetes.io/projected/b136421b-6f82-48c3-9512-5ff3a595ae0a-kube-api-access-pfggf\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcpbd9\" (UID: \"b136421b-6f82-48c3-9512-5ff3a595ae0a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcpbd9" Jul 06 19:31:49.119117 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:49.119021 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b136421b-6f82-48c3-9512-5ff3a595ae0a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcpbd9\" (UID: \"b136421b-6f82-48c3-9512-5ff3a595ae0a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcpbd9" Jul 06 19:31:49.119117 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:49.119048 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b136421b-6f82-48c3-9512-5ff3a595ae0a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcpbd9\" (UID: \"b136421b-6f82-48c3-9512-5ff3a595ae0a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcpbd9" Jul 06 19:31:49.219415 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:49.219390 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfggf\" (UniqueName: \"kubernetes.io/projected/b136421b-6f82-48c3-9512-5ff3a595ae0a-kube-api-access-pfggf\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcpbd9\" (UID: \"b136421b-6f82-48c3-9512-5ff3a595ae0a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcpbd9" Jul 06 19:31:49.219716 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:49.219421 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b136421b-6f82-48c3-9512-5ff3a595ae0a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcpbd9\" (UID: \"b136421b-6f82-48c3-9512-5ff3a595ae0a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcpbd9" Jul 06 19:31:49.219716 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:49.219445 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b136421b-6f82-48c3-9512-5ff3a595ae0a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcpbd9\" (UID: \"b136421b-6f82-48c3-9512-5ff3a595ae0a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcpbd9" Jul 06 19:31:49.219790 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:49.219743 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b136421b-6f82-48c3-9512-5ff3a595ae0a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcpbd9\" (UID: \"b136421b-6f82-48c3-9512-5ff3a595ae0a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcpbd9" Jul 06 19:31:49.219839 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:49.219821 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b136421b-6f82-48c3-9512-5ff3a595ae0a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcpbd9\" (UID: \"b136421b-6f82-48c3-9512-5ff3a595ae0a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcpbd9" Jul 06 19:31:49.273587 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:49.273534 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfggf\" (UniqueName: \"kubernetes.io/projected/b136421b-6f82-48c3-9512-5ff3a595ae0a-kube-api-access-pfggf\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcpbd9\" (UID: \"b136421b-6f82-48c3-9512-5ff3a595ae0a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcpbd9" Jul 06 19:31:49.282341 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:49.282319 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcpbd9" Jul 06 19:31:49.422407 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:49.422270 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcpbd9"] Jul 06 19:31:49.424654 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:31:49.424627 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb136421b_6f82_48c3_9512_5ff3a595ae0a.slice/crio-6cdcdb141ebd9bf95110ca9c37d037715dcf843b4647bfaea27d20b73f92d92a WatchSource:0}: Error finding container 6cdcdb141ebd9bf95110ca9c37d037715dcf843b4647bfaea27d20b73f92d92a: Status 404 returned error can't find the container with id 6cdcdb141ebd9bf95110ca9c37d037715dcf843b4647bfaea27d20b73f92d92a Jul 06 19:31:49.823234 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:49.823206 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-m7qhk"] Jul 06 19:31:49.825421 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:49.825406 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-m7qhk" Jul 06 19:31:49.829163 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:49.829118 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 06 19:31:49.830877 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:49.830857 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 06 19:31:49.832153 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:49.832135 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-rrxpm\"" Jul 06 19:31:49.846515 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:49.846495 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-m7qhk"] Jul 06 19:31:49.925106 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:49.925078 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhkqq\" (UniqueName: \"kubernetes.io/projected/65a0e6d1-9de3-4e11-ad83-1570656e3eae-kube-api-access-rhkqq\") pod \"cert-manager-webhook-6486ffd997-m7qhk\" (UID: \"65a0e6d1-9de3-4e11-ad83-1570656e3eae\") " pod="cert-manager/cert-manager-webhook-6486ffd997-m7qhk" Jul 06 19:31:49.925225 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:49.925132 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/65a0e6d1-9de3-4e11-ad83-1570656e3eae-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-m7qhk\" (UID: \"65a0e6d1-9de3-4e11-ad83-1570656e3eae\") " pod="cert-manager/cert-manager-webhook-6486ffd997-m7qhk" Jul 06 19:31:50.025825 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:50.025801 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhkqq\" (UniqueName: \"kubernetes.io/projected/65a0e6d1-9de3-4e11-ad83-1570656e3eae-kube-api-access-rhkqq\") pod \"cert-manager-webhook-6486ffd997-m7qhk\" (UID: \"65a0e6d1-9de3-4e11-ad83-1570656e3eae\") " pod="cert-manager/cert-manager-webhook-6486ffd997-m7qhk" Jul 06 19:31:50.025915 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:50.025864 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/65a0e6d1-9de3-4e11-ad83-1570656e3eae-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-m7qhk\" (UID: \"65a0e6d1-9de3-4e11-ad83-1570656e3eae\") " pod="cert-manager/cert-manager-webhook-6486ffd997-m7qhk" Jul 06 19:31:50.035490 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:50.035467 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/65a0e6d1-9de3-4e11-ad83-1570656e3eae-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-m7qhk\" (UID: \"65a0e6d1-9de3-4e11-ad83-1570656e3eae\") " pod="cert-manager/cert-manager-webhook-6486ffd997-m7qhk" Jul 06 19:31:50.035600 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:50.035583 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhkqq\" (UniqueName: \"kubernetes.io/projected/65a0e6d1-9de3-4e11-ad83-1570656e3eae-kube-api-access-rhkqq\") pod \"cert-manager-webhook-6486ffd997-m7qhk\" (UID: \"65a0e6d1-9de3-4e11-ad83-1570656e3eae\") " pod="cert-manager/cert-manager-webhook-6486ffd997-m7qhk" Jul 06 19:31:50.146638 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:50.146582 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-m7qhk" Jul 06 19:31:50.153807 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:50.153787 2575 generic.go:356] "Generic (PLEG): container finished" podID="b136421b-6f82-48c3-9512-5ff3a595ae0a" containerID="dc1c78da92403f06d5882cee0f84ff2b4d8da4f56cc2f80b8a1e7fbd8f8c9735" exitCode=0 Jul 06 19:31:50.153894 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:50.153877 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcpbd9" event={"ID":"b136421b-6f82-48c3-9512-5ff3a595ae0a","Type":"ContainerDied","Data":"dc1c78da92403f06d5882cee0f84ff2b4d8da4f56cc2f80b8a1e7fbd8f8c9735"} Jul 06 19:31:50.153945 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:50.153928 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcpbd9" event={"ID":"b136421b-6f82-48c3-9512-5ff3a595ae0a","Type":"ContainerStarted","Data":"6cdcdb141ebd9bf95110ca9c37d037715dcf843b4647bfaea27d20b73f92d92a"} Jul 06 19:31:50.279849 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:50.279827 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-m7qhk"] Jul 06 19:31:50.282354 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:31:50.281459 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65a0e6d1_9de3_4e11_ad83_1570656e3eae.slice/crio-ee8fe52b3418c14e26c1304ca26b37dc335a699004458f5d6a45032a5a0ecc70 WatchSource:0}: Error finding container ee8fe52b3418c14e26c1304ca26b37dc335a699004458f5d6a45032a5a0ecc70: Status 404 returned error can't find the container with id ee8fe52b3418c14e26c1304ca26b37dc335a699004458f5d6a45032a5a0ecc70 Jul 06 19:31:51.161159 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:51.161115 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-m7qhk" event={"ID":"65a0e6d1-9de3-4e11-ad83-1570656e3eae","Type":"ContainerStarted","Data":"ee8fe52b3418c14e26c1304ca26b37dc335a699004458f5d6a45032a5a0ecc70"} Jul 06 19:31:52.169223 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:52.169112 2575 generic.go:356] "Generic (PLEG): container finished" podID="b136421b-6f82-48c3-9512-5ff3a595ae0a" containerID="a555f4ecc5e27c96233fd0f02480d5b52654a4c24148f7a34c5535229f56df8b" exitCode=0 Jul 06 19:31:52.169223 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:52.169209 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcpbd9" event={"ID":"b136421b-6f82-48c3-9512-5ff3a595ae0a","Type":"ContainerDied","Data":"a555f4ecc5e27c96233fd0f02480d5b52654a4c24148f7a34c5535229f56df8b"} Jul 06 19:31:53.176400 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:53.176368 2575 generic.go:356] "Generic (PLEG): container finished" podID="b136421b-6f82-48c3-9512-5ff3a595ae0a" containerID="de149a4ca13ffe829bf51a3f710a9d8f1c17a2e6d7964bdc4f0eace951e4694a" exitCode=0 Jul 06 19:31:53.176802 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:53.176407 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcpbd9" event={"ID":"b136421b-6f82-48c3-9512-5ff3a595ae0a","Type":"ContainerDied","Data":"de149a4ca13ffe829bf51a3f710a9d8f1c17a2e6d7964bdc4f0eace951e4694a"} Jul 06 19:31:54.182925 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:54.182885 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-m7qhk" event={"ID":"65a0e6d1-9de3-4e11-ad83-1570656e3eae","Type":"ContainerStarted","Data":"ede94ecd92953b178b924482cff7ce218a3832eb6a1c0c7c03b28bd61c09ea63"} Jul 06 19:31:54.211810 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:54.210679 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-m7qhk" podStartSLOduration=1.6867585360000001 podStartE2EDuration="5.210656625s" totalImagesPullingTime="3.523898089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:31:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:31:54.209168155 +0000 UTC m=+323.029256868" watchObservedRunningTime="2026-07-06 19:31:54.210656625 +0000 UTC m=+323.030745339" Jul 06 19:31:54.304127 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:54.304105 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcpbd9" Jul 06 19:31:54.468697 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:54.468627 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b136421b-6f82-48c3-9512-5ff3a595ae0a-bundle\") pod \"b136421b-6f82-48c3-9512-5ff3a595ae0a\" (UID: \"b136421b-6f82-48c3-9512-5ff3a595ae0a\") " Jul 06 19:31:54.468697 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:54.468691 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b136421b-6f82-48c3-9512-5ff3a595ae0a-util\") pod \"b136421b-6f82-48c3-9512-5ff3a595ae0a\" (UID: \"b136421b-6f82-48c3-9512-5ff3a595ae0a\") " Jul 06 19:31:54.468870 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:54.468779 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfggf\" (UniqueName: \"kubernetes.io/projected/b136421b-6f82-48c3-9512-5ff3a595ae0a-kube-api-access-pfggf\") pod \"b136421b-6f82-48c3-9512-5ff3a595ae0a\" (UID: \"b136421b-6f82-48c3-9512-5ff3a595ae0a\") " Jul 06 19:31:54.469021 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:54.468997 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b136421b-6f82-48c3-9512-5ff3a595ae0a-bundle" (OuterVolumeSpecName: "bundle") pod "b136421b-6f82-48c3-9512-5ff3a595ae0a" (UID: "b136421b-6f82-48c3-9512-5ff3a595ae0a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:31:54.470820 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:54.470791 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b136421b-6f82-48c3-9512-5ff3a595ae0a-kube-api-access-pfggf" (OuterVolumeSpecName: "kube-api-access-pfggf") pod "b136421b-6f82-48c3-9512-5ff3a595ae0a" (UID: "b136421b-6f82-48c3-9512-5ff3a595ae0a"). InnerVolumeSpecName "kube-api-access-pfggf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:31:54.474012 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:54.473984 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b136421b-6f82-48c3-9512-5ff3a595ae0a-util" (OuterVolumeSpecName: "util") pod "b136421b-6f82-48c3-9512-5ff3a595ae0a" (UID: "b136421b-6f82-48c3-9512-5ff3a595ae0a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:31:54.569610 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:54.569576 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfggf\" (UniqueName: \"kubernetes.io/projected/b136421b-6f82-48c3-9512-5ff3a595ae0a-kube-api-access-pfggf\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:31:54.569610 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:54.569608 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b136421b-6f82-48c3-9512-5ff3a595ae0a-bundle\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:31:54.569610 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:54.569617 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b136421b-6f82-48c3-9512-5ff3a595ae0a-util\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:31:55.188854 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:55.188823 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcpbd9" Jul 06 19:31:55.188854 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:55.188843 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcpbd9" event={"ID":"b136421b-6f82-48c3-9512-5ff3a595ae0a","Type":"ContainerDied","Data":"6cdcdb141ebd9bf95110ca9c37d037715dcf843b4647bfaea27d20b73f92d92a"} Jul 06 19:31:55.189366 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:55.188879 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cdcdb141ebd9bf95110ca9c37d037715dcf843b4647bfaea27d20b73f92d92a" Jul 06 19:31:55.189366 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:31:55.188925 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-m7qhk" Jul 06 19:32:01.195987 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:01.195958 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-m7qhk" Jul 06 19:32:01.643222 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:01.643195 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2k5tq"] Jul 06 19:32:01.643525 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:01.643513 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b136421b-6f82-48c3-9512-5ff3a595ae0a" containerName="pull" Jul 06 19:32:01.643567 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:01.643526 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b136421b-6f82-48c3-9512-5ff3a595ae0a" containerName="pull" Jul 06 19:32:01.643567 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:01.643545 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b136421b-6f82-48c3-9512-5ff3a595ae0a" containerName="util" Jul 06 19:32:01.643567 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:01.643552 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b136421b-6f82-48c3-9512-5ff3a595ae0a" containerName="util" Jul 06 19:32:01.643567 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:01.643559 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b136421b-6f82-48c3-9512-5ff3a595ae0a" containerName="extract" Jul 06 19:32:01.643567 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:01.643564 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b136421b-6f82-48c3-9512-5ff3a595ae0a" containerName="extract" Jul 06 19:32:01.643713 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:01.643623 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b136421b-6f82-48c3-9512-5ff3a595ae0a" containerName="extract" Jul 06 19:32:01.645614 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:01.645600 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2k5tq" Jul 06 19:32:01.648240 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:01.648210 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 06 19:32:01.648357 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:01.648284 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:32:01.649255 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:01.649240 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-6bt7b\"" Jul 06 19:32:01.656969 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:01.656947 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2k5tq"] Jul 06 19:32:01.829072 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:01.829044 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f4de39b-3797-4653-99bd-150dd6dd9955-tmp\") pod \"openshift-lws-operator-fd8ccff4c-2k5tq\" (UID: \"9f4de39b-3797-4653-99bd-150dd6dd9955\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2k5tq" Jul 06 19:32:01.829232 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:01.829124 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhmnz\" (UniqueName: \"kubernetes.io/projected/9f4de39b-3797-4653-99bd-150dd6dd9955-kube-api-access-hhmnz\") pod \"openshift-lws-operator-fd8ccff4c-2k5tq\" (UID: \"9f4de39b-3797-4653-99bd-150dd6dd9955\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2k5tq" Jul 06 19:32:01.929526 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:01.929467 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f4de39b-3797-4653-99bd-150dd6dd9955-tmp\") pod \"openshift-lws-operator-fd8ccff4c-2k5tq\" (UID: \"9f4de39b-3797-4653-99bd-150dd6dd9955\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2k5tq" Jul 06 19:32:01.929526 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:01.929506 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhmnz\" (UniqueName: \"kubernetes.io/projected/9f4de39b-3797-4653-99bd-150dd6dd9955-kube-api-access-hhmnz\") pod \"openshift-lws-operator-fd8ccff4c-2k5tq\" (UID: \"9f4de39b-3797-4653-99bd-150dd6dd9955\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2k5tq" Jul 06 19:32:01.929784 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:01.929766 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f4de39b-3797-4653-99bd-150dd6dd9955-tmp\") pod \"openshift-lws-operator-fd8ccff4c-2k5tq\" (UID: \"9f4de39b-3797-4653-99bd-150dd6dd9955\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2k5tq" Jul 06 19:32:01.943958 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:01.943934 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhmnz\" (UniqueName: \"kubernetes.io/projected/9f4de39b-3797-4653-99bd-150dd6dd9955-kube-api-access-hhmnz\") pod \"openshift-lws-operator-fd8ccff4c-2k5tq\" (UID: \"9f4de39b-3797-4653-99bd-150dd6dd9955\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2k5tq" Jul 06 19:32:01.954746 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:01.954727 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2k5tq" Jul 06 19:32:02.097079 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:02.097054 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2k5tq"] Jul 06 19:32:02.098976 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:32:02.098947 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f4de39b_3797_4653_99bd_150dd6dd9955.slice/crio-ca809706b998751be196a407555e6d28cdf610cc6d33544d5a5ec7f2ade8b6ec WatchSource:0}: Error finding container ca809706b998751be196a407555e6d28cdf610cc6d33544d5a5ec7f2ade8b6ec: Status 404 returned error can't find the container with id ca809706b998751be196a407555e6d28cdf610cc6d33544d5a5ec7f2ade8b6ec Jul 06 19:32:02.148556 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:02.148164 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-xr8z6"] Jul 06 19:32:02.152899 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:02.152876 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-xr8z6" Jul 06 19:32:02.156374 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:02.156359 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-pnmbx\"" Jul 06 19:32:02.164971 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:02.164950 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-xr8z6"] Jul 06 19:32:02.222588 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:02.222538 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2k5tq" event={"ID":"9f4de39b-3797-4653-99bd-150dd6dd9955","Type":"ContainerStarted","Data":"ca809706b998751be196a407555e6d28cdf610cc6d33544d5a5ec7f2ade8b6ec"} Jul 06 19:32:02.232934 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:02.232912 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1545a6ee-a6e1-440f-b64a-df048d2b793f-bound-sa-token\") pod \"cert-manager-55b4d954d5-xr8z6\" (UID: \"1545a6ee-a6e1-440f-b64a-df048d2b793f\") " pod="cert-manager/cert-manager-55b4d954d5-xr8z6" Jul 06 19:32:02.233008 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:02.232971 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n4rp\" (UniqueName: \"kubernetes.io/projected/1545a6ee-a6e1-440f-b64a-df048d2b793f-kube-api-access-7n4rp\") pod \"cert-manager-55b4d954d5-xr8z6\" (UID: \"1545a6ee-a6e1-440f-b64a-df048d2b793f\") " pod="cert-manager/cert-manager-55b4d954d5-xr8z6" Jul 06 19:32:02.333963 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:02.333940 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1545a6ee-a6e1-440f-b64a-df048d2b793f-bound-sa-token\") pod \"cert-manager-55b4d954d5-xr8z6\" (UID: \"1545a6ee-a6e1-440f-b64a-df048d2b793f\") " pod="cert-manager/cert-manager-55b4d954d5-xr8z6" Jul 06 19:32:02.334051 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:02.333983 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n4rp\" (UniqueName: \"kubernetes.io/projected/1545a6ee-a6e1-440f-b64a-df048d2b793f-kube-api-access-7n4rp\") pod \"cert-manager-55b4d954d5-xr8z6\" (UID: \"1545a6ee-a6e1-440f-b64a-df048d2b793f\") " pod="cert-manager/cert-manager-55b4d954d5-xr8z6" Jul 06 19:32:02.345784 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:02.345756 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1545a6ee-a6e1-440f-b64a-df048d2b793f-bound-sa-token\") pod \"cert-manager-55b4d954d5-xr8z6\" (UID: \"1545a6ee-a6e1-440f-b64a-df048d2b793f\") " pod="cert-manager/cert-manager-55b4d954d5-xr8z6" Jul 06 19:32:02.345872 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:02.345854 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n4rp\" (UniqueName: \"kubernetes.io/projected/1545a6ee-a6e1-440f-b64a-df048d2b793f-kube-api-access-7n4rp\") pod \"cert-manager-55b4d954d5-xr8z6\" (UID: \"1545a6ee-a6e1-440f-b64a-df048d2b793f\") " pod="cert-manager/cert-manager-55b4d954d5-xr8z6" Jul 06 19:32:02.462386 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:02.462364 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-xr8z6" Jul 06 19:32:02.589242 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:02.589219 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-xr8z6"] Jul 06 19:32:02.591227 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:32:02.591202 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1545a6ee_a6e1_440f_b64a_df048d2b793f.slice/crio-3ee8cf9f18ace79d3197827526c7ae8ad1d4e858f2a8fd90ed92abe7f5a79ced WatchSource:0}: Error finding container 3ee8cf9f18ace79d3197827526c7ae8ad1d4e858f2a8fd90ed92abe7f5a79ced: Status 404 returned error can't find the container with id 3ee8cf9f18ace79d3197827526c7ae8ad1d4e858f2a8fd90ed92abe7f5a79ced Jul 06 19:32:03.229895 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:03.229863 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-xr8z6" event={"ID":"1545a6ee-a6e1-440f-b64a-df048d2b793f","Type":"ContainerStarted","Data":"0fc7d1cf45d24949eba4d30baf91492cc9ad4dab4f91c871d028a9c8ce5a75da"} Jul 06 19:32:03.230326 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:03.229908 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-xr8z6" event={"ID":"1545a6ee-a6e1-440f-b64a-df048d2b793f","Type":"ContainerStarted","Data":"3ee8cf9f18ace79d3197827526c7ae8ad1d4e858f2a8fd90ed92abe7f5a79ced"} Jul 06 19:32:03.250942 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:03.250622 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-xr8z6" podStartSLOduration=1.250606012 podStartE2EDuration="1.250606012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:32:03.250107975 +0000 UTC m=+332.070196688" watchObservedRunningTime="2026-07-06 19:32:03.250606012 +0000 UTC m=+332.070694725" Jul 06 19:32:04.237624 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:04.237582 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2k5tq" event={"ID":"9f4de39b-3797-4653-99bd-150dd6dd9955","Type":"ContainerStarted","Data":"66faec3a2449bd85634d3b320d89f5fecc32382a8effeaa356c9f22e5fadff11"} Jul 06 19:32:04.258815 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:04.258757 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2k5tq" podStartSLOduration=1.6469852390000002 podStartE2EDuration="3.258739446s" totalImagesPullingTime="1.611754207s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:32:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:32:04.257235144 +0000 UTC m=+333.077323853" watchObservedRunningTime="2026-07-06 19:32:04.258739446 +0000 UTC m=+333.078828160" Jul 06 19:32:07.891195 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:07.891156 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzhv7"] Jul 06 19:32:07.929747 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:07.929716 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzhv7"] Jul 06 19:32:07.929900 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:07.929823 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzhv7" Jul 06 19:32:07.932728 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:07.932706 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 06 19:32:07.932845 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:07.932707 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-c7vvn\"" Jul 06 19:32:07.933784 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:07.933770 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 06 19:32:07.975378 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:07.975353 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62f746f7-42dc-4b40-ae89-6558cc62331a-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzhv7\" (UID: \"62f746f7-42dc-4b40-ae89-6558cc62331a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzhv7" Jul 06 19:32:07.975475 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:07.975399 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62f746f7-42dc-4b40-ae89-6558cc62331a-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzhv7\" (UID: \"62f746f7-42dc-4b40-ae89-6558cc62331a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzhv7" Jul 06 19:32:07.975475 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:07.975427 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5nnk\" (UniqueName: \"kubernetes.io/projected/62f746f7-42dc-4b40-ae89-6558cc62331a-kube-api-access-b5nnk\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzhv7\" (UID: \"62f746f7-42dc-4b40-ae89-6558cc62331a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzhv7" Jul 06 19:32:08.076601 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:08.076576 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62f746f7-42dc-4b40-ae89-6558cc62331a-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzhv7\" (UID: \"62f746f7-42dc-4b40-ae89-6558cc62331a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzhv7" Jul 06 19:32:08.076685 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:08.076613 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62f746f7-42dc-4b40-ae89-6558cc62331a-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzhv7\" (UID: \"62f746f7-42dc-4b40-ae89-6558cc62331a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzhv7" Jul 06 19:32:08.076685 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:08.076633 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5nnk\" (UniqueName: \"kubernetes.io/projected/62f746f7-42dc-4b40-ae89-6558cc62331a-kube-api-access-b5nnk\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzhv7\" (UID: \"62f746f7-42dc-4b40-ae89-6558cc62331a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzhv7" Jul 06 19:32:08.076933 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:08.076917 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62f746f7-42dc-4b40-ae89-6558cc62331a-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzhv7\" (UID: \"62f746f7-42dc-4b40-ae89-6558cc62331a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzhv7" Jul 06 19:32:08.076970 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:08.076955 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62f746f7-42dc-4b40-ae89-6558cc62331a-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzhv7\" (UID: \"62f746f7-42dc-4b40-ae89-6558cc62331a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzhv7" Jul 06 19:32:08.097740 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:08.097712 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5nnk\" (UniqueName: \"kubernetes.io/projected/62f746f7-42dc-4b40-ae89-6558cc62331a-kube-api-access-b5nnk\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzhv7\" (UID: \"62f746f7-42dc-4b40-ae89-6558cc62331a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzhv7" Jul 06 19:32:08.239282 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:08.239229 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzhv7" Jul 06 19:32:08.408544 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:08.408521 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzhv7"] Jul 06 19:32:08.410600 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:32:08.410575 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62f746f7_42dc_4b40_ae89_6558cc62331a.slice/crio-1be5eb152baceb5223b95ecf21d905c1c2c00992134790731f3d8bafda4e420b WatchSource:0}: Error finding container 1be5eb152baceb5223b95ecf21d905c1c2c00992134790731f3d8bafda4e420b: Status 404 returned error can't find the container with id 1be5eb152baceb5223b95ecf21d905c1c2c00992134790731f3d8bafda4e420b Jul 06 19:32:09.262367 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:09.262335 2575 generic.go:356] "Generic (PLEG): container finished" podID="62f746f7-42dc-4b40-ae89-6558cc62331a" containerID="dfb969a681652f04b05d311719c67f6eb730c6839962f20b56c952c61302b1b7" exitCode=0 Jul 06 19:32:09.262761 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:09.262417 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzhv7" event={"ID":"62f746f7-42dc-4b40-ae89-6558cc62331a","Type":"ContainerDied","Data":"dfb969a681652f04b05d311719c67f6eb730c6839962f20b56c952c61302b1b7"} Jul 06 19:32:09.262761 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:09.262465 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzhv7" event={"ID":"62f746f7-42dc-4b40-ae89-6558cc62331a","Type":"ContainerStarted","Data":"1be5eb152baceb5223b95ecf21d905c1c2c00992134790731f3d8bafda4e420b"} Jul 06 19:32:11.273522 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:11.273486 2575 generic.go:356] "Generic (PLEG): container finished" podID="62f746f7-42dc-4b40-ae89-6558cc62331a" containerID="6fa4eac58ae0d7e64f439d277fcbcbe0f3de85366aec24a73b966e306e7d6fc5" exitCode=0 Jul 06 19:32:11.273870 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:11.273531 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzhv7" event={"ID":"62f746f7-42dc-4b40-ae89-6558cc62331a","Type":"ContainerDied","Data":"6fa4eac58ae0d7e64f439d277fcbcbe0f3de85366aec24a73b966e306e7d6fc5"} Jul 06 19:32:12.280383 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:12.280345 2575 generic.go:356] "Generic (PLEG): container finished" podID="62f746f7-42dc-4b40-ae89-6558cc62331a" containerID="0a9661e4d8d50d3fe9dd7066bd61412a21019baa941411c2a5885919d40d339a" exitCode=0 Jul 06 19:32:12.280797 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:12.280404 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzhv7" event={"ID":"62f746f7-42dc-4b40-ae89-6558cc62331a","Type":"ContainerDied","Data":"0a9661e4d8d50d3fe9dd7066bd61412a21019baa941411c2a5885919d40d339a"} Jul 06 19:32:13.419237 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:13.419212 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzhv7" Jul 06 19:32:13.521068 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:13.521039 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62f746f7-42dc-4b40-ae89-6558cc62331a-util\") pod \"62f746f7-42dc-4b40-ae89-6558cc62331a\" (UID: \"62f746f7-42dc-4b40-ae89-6558cc62331a\") " Jul 06 19:32:13.521224 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:13.521119 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62f746f7-42dc-4b40-ae89-6558cc62331a-bundle\") pod \"62f746f7-42dc-4b40-ae89-6558cc62331a\" (UID: \"62f746f7-42dc-4b40-ae89-6558cc62331a\") " Jul 06 19:32:13.521224 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:13.521142 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5nnk\" (UniqueName: \"kubernetes.io/projected/62f746f7-42dc-4b40-ae89-6558cc62331a-kube-api-access-b5nnk\") pod \"62f746f7-42dc-4b40-ae89-6558cc62331a\" (UID: \"62f746f7-42dc-4b40-ae89-6558cc62331a\") " Jul 06 19:32:13.521804 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:13.521774 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62f746f7-42dc-4b40-ae89-6558cc62331a-bundle" (OuterVolumeSpecName: "bundle") pod "62f746f7-42dc-4b40-ae89-6558cc62331a" (UID: "62f746f7-42dc-4b40-ae89-6558cc62331a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:32:13.523244 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:13.523215 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62f746f7-42dc-4b40-ae89-6558cc62331a-kube-api-access-b5nnk" (OuterVolumeSpecName: "kube-api-access-b5nnk") pod "62f746f7-42dc-4b40-ae89-6558cc62331a" (UID: "62f746f7-42dc-4b40-ae89-6558cc62331a"). InnerVolumeSpecName "kube-api-access-b5nnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:32:13.526368 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:13.526348 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62f746f7-42dc-4b40-ae89-6558cc62331a-util" (OuterVolumeSpecName: "util") pod "62f746f7-42dc-4b40-ae89-6558cc62331a" (UID: "62f746f7-42dc-4b40-ae89-6558cc62331a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:32:13.622227 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:13.622205 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62f746f7-42dc-4b40-ae89-6558cc62331a-util\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:32:13.622227 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:13.622224 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62f746f7-42dc-4b40-ae89-6558cc62331a-bundle\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:32:13.622356 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:13.622234 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5nnk\" (UniqueName: \"kubernetes.io/projected/62f746f7-42dc-4b40-ae89-6558cc62331a-kube-api-access-b5nnk\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:32:14.292381 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:14.292350 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzhv7" Jul 06 19:32:14.292381 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:14.292369 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflzhv7" event={"ID":"62f746f7-42dc-4b40-ae89-6558cc62331a","Type":"ContainerDied","Data":"1be5eb152baceb5223b95ecf21d905c1c2c00992134790731f3d8bafda4e420b"} Jul 06 19:32:14.292587 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:14.292406 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1be5eb152baceb5223b95ecf21d905c1c2c00992134790731f3d8bafda4e420b" Jul 06 19:32:22.661735 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:22.661702 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5586496c87-l267l"] Jul 06 19:32:22.662205 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:22.662061 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62f746f7-42dc-4b40-ae89-6558cc62331a" containerName="extract" Jul 06 19:32:22.662205 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:22.662072 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="62f746f7-42dc-4b40-ae89-6558cc62331a" containerName="extract" Jul 06 19:32:22.662205 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:22.662081 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62f746f7-42dc-4b40-ae89-6558cc62331a" containerName="util" Jul 06 19:32:22.662205 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:22.662086 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="62f746f7-42dc-4b40-ae89-6558cc62331a" containerName="util" Jul 06 19:32:22.662205 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:22.662092 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62f746f7-42dc-4b40-ae89-6558cc62331a" containerName="pull" Jul 06 19:32:22.662205 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:22.662097 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="62f746f7-42dc-4b40-ae89-6558cc62331a" containerName="pull" Jul 06 19:32:22.662205 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:22.662156 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="62f746f7-42dc-4b40-ae89-6558cc62331a" containerName="extract" Jul 06 19:32:22.666572 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:22.666550 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-l267l" Jul 06 19:32:22.675798 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:22.675778 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 06 19:32:22.681206 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:22.681165 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 06 19:32:22.681330 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:22.681165 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 06 19:32:22.681330 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:22.681166 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 06 19:32:22.681330 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:22.681166 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-7kjrq\"" Jul 06 19:32:22.684705 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:22.684680 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5586496c87-l267l"] Jul 06 19:32:22.797576 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:22.797551 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de811e8a-5b85-4707-abe7-79107ab0a37a-tmp\") pod \"opendatahub-operator-controller-manager-5586496c87-l267l\" (UID: \"de811e8a-5b85-4707-abe7-79107ab0a37a\") " pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-l267l" Jul 06 19:32:22.797735 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:22.797580 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d7n4\" (UniqueName: \"kubernetes.io/projected/de811e8a-5b85-4707-abe7-79107ab0a37a-kube-api-access-8d7n4\") pod \"opendatahub-operator-controller-manager-5586496c87-l267l\" (UID: \"de811e8a-5b85-4707-abe7-79107ab0a37a\") " pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-l267l" Jul 06 19:32:22.797735 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:22.797666 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/de811e8a-5b85-4707-abe7-79107ab0a37a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5586496c87-l267l\" (UID: \"de811e8a-5b85-4707-abe7-79107ab0a37a\") " pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-l267l" Jul 06 19:32:22.797735 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:22.797725 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/de811e8a-5b85-4707-abe7-79107ab0a37a-manifests\") pod \"opendatahub-operator-controller-manager-5586496c87-l267l\" (UID: \"de811e8a-5b85-4707-abe7-79107ab0a37a\") " pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-l267l" Jul 06 19:32:22.797841 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:22.797746 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/de811e8a-5b85-4707-abe7-79107ab0a37a-webhook-cert\") pod \"opendatahub-operator-controller-manager-5586496c87-l267l\" (UID: \"de811e8a-5b85-4707-abe7-79107ab0a37a\") " pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-l267l" Jul 06 19:32:22.898255 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:22.898175 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/de811e8a-5b85-4707-abe7-79107ab0a37a-manifests\") pod \"opendatahub-operator-controller-manager-5586496c87-l267l\" (UID: \"de811e8a-5b85-4707-abe7-79107ab0a37a\") " pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-l267l" Jul 06 19:32:22.898255 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:22.898233 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/de811e8a-5b85-4707-abe7-79107ab0a37a-webhook-cert\") pod \"opendatahub-operator-controller-manager-5586496c87-l267l\" (UID: \"de811e8a-5b85-4707-abe7-79107ab0a37a\") " pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-l267l" Jul 06 19:32:22.898428 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:22.898271 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de811e8a-5b85-4707-abe7-79107ab0a37a-tmp\") pod \"opendatahub-operator-controller-manager-5586496c87-l267l\" (UID: \"de811e8a-5b85-4707-abe7-79107ab0a37a\") " pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-l267l" Jul 06 19:32:22.898428 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:22.898296 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d7n4\" (UniqueName: \"kubernetes.io/projected/de811e8a-5b85-4707-abe7-79107ab0a37a-kube-api-access-8d7n4\") pod \"opendatahub-operator-controller-manager-5586496c87-l267l\" (UID: \"de811e8a-5b85-4707-abe7-79107ab0a37a\") " pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-l267l" Jul 06 19:32:22.898428 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:22.898392 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/de811e8a-5b85-4707-abe7-79107ab0a37a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5586496c87-l267l\" (UID: \"de811e8a-5b85-4707-abe7-79107ab0a37a\") " pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-l267l" Jul 06 19:32:22.899329 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:22.899308 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/de811e8a-5b85-4707-abe7-79107ab0a37a-manifests\") pod \"opendatahub-operator-controller-manager-5586496c87-l267l\" (UID: \"de811e8a-5b85-4707-abe7-79107ab0a37a\") " pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-l267l" Jul 06 19:32:22.901031 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:22.901010 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de811e8a-5b85-4707-abe7-79107ab0a37a-tmp\") pod \"opendatahub-operator-controller-manager-5586496c87-l267l\" (UID: \"de811e8a-5b85-4707-abe7-79107ab0a37a\") " pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-l267l" Jul 06 19:32:22.901340 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:22.901325 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/de811e8a-5b85-4707-abe7-79107ab0a37a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5586496c87-l267l\" (UID: \"de811e8a-5b85-4707-abe7-79107ab0a37a\") " pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-l267l" Jul 06 19:32:22.901395 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:22.901347 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/de811e8a-5b85-4707-abe7-79107ab0a37a-webhook-cert\") pod \"opendatahub-operator-controller-manager-5586496c87-l267l\" (UID: \"de811e8a-5b85-4707-abe7-79107ab0a37a\") " pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-l267l" Jul 06 19:32:22.911697 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:22.911669 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d7n4\" (UniqueName: \"kubernetes.io/projected/de811e8a-5b85-4707-abe7-79107ab0a37a-kube-api-access-8d7n4\") pod \"opendatahub-operator-controller-manager-5586496c87-l267l\" (UID: \"de811e8a-5b85-4707-abe7-79107ab0a37a\") " pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-l267l" Jul 06 19:32:22.978003 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:22.977952 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-l267l" Jul 06 19:32:23.130462 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:23.130441 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5586496c87-l267l"] Jul 06 19:32:23.131817 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:32:23.131791 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde811e8a_5b85_4707_abe7_79107ab0a37a.slice/crio-08725c9398773d606b6f18d2b7af6ed81ddad97a226d43a92dd833dc1e91b3bc WatchSource:0}: Error finding container 08725c9398773d606b6f18d2b7af6ed81ddad97a226d43a92dd833dc1e91b3bc: Status 404 returned error can't find the container with id 08725c9398773d606b6f18d2b7af6ed81ddad97a226d43a92dd833dc1e91b3bc Jul 06 19:32:23.336754 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:23.336729 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-l267l" event={"ID":"de811e8a-5b85-4707-abe7-79107ab0a37a","Type":"ContainerStarted","Data":"08725c9398773d606b6f18d2b7af6ed81ddad97a226d43a92dd833dc1e91b3bc"} Jul 06 19:32:26.354318 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:26.354248 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-l267l" event={"ID":"de811e8a-5b85-4707-abe7-79107ab0a37a","Type":"ContainerStarted","Data":"1b24e0d91318987228ff616e33a5644b6aaa63df80768dc8f38f0ec3aa2bc601"} Jul 06 19:32:27.361015 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:27.360982 2575 generic.go:356] "Generic (PLEG): container finished" podID="de811e8a-5b85-4707-abe7-79107ab0a37a" containerID="1b24e0d91318987228ff616e33a5644b6aaa63df80768dc8f38f0ec3aa2bc601" exitCode=0 Jul 06 19:32:27.361379 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:27.361038 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-l267l" event={"ID":"de811e8a-5b85-4707-abe7-79107ab0a37a","Type":"ContainerDied","Data":"1b24e0d91318987228ff616e33a5644b6aaa63df80768dc8f38f0ec3aa2bc601"} Jul 06 19:32:28.367812 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:28.367777 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-l267l" event={"ID":"de811e8a-5b85-4707-abe7-79107ab0a37a","Type":"ContainerStarted","Data":"2a93ab5ba80dc4bcff47229dd64bf581d0e0f515cc52aeb9860a4a2c8f584d9d"} Jul 06 19:32:28.368198 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:28.367941 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-l267l" Jul 06 19:32:39.376248 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:39.376210 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-l267l" Jul 06 19:32:39.416288 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:39.416238 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-l267l" podStartSLOduration=13.044402097 podStartE2EDuration="17.41622315s" totalImagesPullingTime="3.356905739s" totalInitContainerRuntime="1.014915314s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:32:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:32:28.402560515 +0000 UTC m=+357.222649229" watchObservedRunningTime="2026-07-06 19:32:39.41622315 +0000 UTC m=+368.236311861" Jul 06 19:32:42.616431 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:42.616396 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7c47f5985b-pczqp"] Jul 06 19:32:42.622291 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:42.622263 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c47f5985b-pczqp" Jul 06 19:32:42.626096 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:42.626068 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 06 19:32:42.626247 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:42.626097 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 06 19:32:42.634718 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:42.634691 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c47f5985b-pczqp"] Jul 06 19:32:42.766066 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:42.766035 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b77b8d39-d01b-409d-bbbc-1e30d9200e1c-tmp\") pod \"kube-auth-proxy-7c47f5985b-pczqp\" (UID: \"b77b8d39-d01b-409d-bbbc-1e30d9200e1c\") " pod="openshift-ingress/kube-auth-proxy-7c47f5985b-pczqp" Jul 06 19:32:42.766251 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:42.766073 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b77b8d39-d01b-409d-bbbc-1e30d9200e1c-tls-certs\") pod \"kube-auth-proxy-7c47f5985b-pczqp\" (UID: \"b77b8d39-d01b-409d-bbbc-1e30d9200e1c\") " pod="openshift-ingress/kube-auth-proxy-7c47f5985b-pczqp" Jul 06 19:32:42.766251 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:42.766090 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcl49\" (UniqueName: \"kubernetes.io/projected/b77b8d39-d01b-409d-bbbc-1e30d9200e1c-kube-api-access-bcl49\") pod \"kube-auth-proxy-7c47f5985b-pczqp\" (UID: \"b77b8d39-d01b-409d-bbbc-1e30d9200e1c\") " pod="openshift-ingress/kube-auth-proxy-7c47f5985b-pczqp" Jul 06 19:32:42.867598 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:42.867530 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b77b8d39-d01b-409d-bbbc-1e30d9200e1c-tmp\") pod \"kube-auth-proxy-7c47f5985b-pczqp\" (UID: \"b77b8d39-d01b-409d-bbbc-1e30d9200e1c\") " pod="openshift-ingress/kube-auth-proxy-7c47f5985b-pczqp" Jul 06 19:32:42.867598 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:42.867566 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b77b8d39-d01b-409d-bbbc-1e30d9200e1c-tls-certs\") pod \"kube-auth-proxy-7c47f5985b-pczqp\" (UID: \"b77b8d39-d01b-409d-bbbc-1e30d9200e1c\") " pod="openshift-ingress/kube-auth-proxy-7c47f5985b-pczqp" Jul 06 19:32:42.867598 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:42.867592 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcl49\" (UniqueName: \"kubernetes.io/projected/b77b8d39-d01b-409d-bbbc-1e30d9200e1c-kube-api-access-bcl49\") pod \"kube-auth-proxy-7c47f5985b-pczqp\" (UID: \"b77b8d39-d01b-409d-bbbc-1e30d9200e1c\") " pod="openshift-ingress/kube-auth-proxy-7c47f5985b-pczqp" Jul 06 19:32:42.869814 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:42.869793 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b77b8d39-d01b-409d-bbbc-1e30d9200e1c-tmp\") pod \"kube-auth-proxy-7c47f5985b-pczqp\" (UID: \"b77b8d39-d01b-409d-bbbc-1e30d9200e1c\") " pod="openshift-ingress/kube-auth-proxy-7c47f5985b-pczqp" Jul 06 19:32:42.870112 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:42.870092 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b77b8d39-d01b-409d-bbbc-1e30d9200e1c-tls-certs\") pod \"kube-auth-proxy-7c47f5985b-pczqp\" (UID: \"b77b8d39-d01b-409d-bbbc-1e30d9200e1c\") " pod="openshift-ingress/kube-auth-proxy-7c47f5985b-pczqp" Jul 06 19:32:42.886459 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:42.886430 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcl49\" (UniqueName: \"kubernetes.io/projected/b77b8d39-d01b-409d-bbbc-1e30d9200e1c-kube-api-access-bcl49\") pod \"kube-auth-proxy-7c47f5985b-pczqp\" (UID: \"b77b8d39-d01b-409d-bbbc-1e30d9200e1c\") " pod="openshift-ingress/kube-auth-proxy-7c47f5985b-pczqp" Jul 06 19:32:42.932341 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:42.932302 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c47f5985b-pczqp" Jul 06 19:32:42.974629 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:42.974591 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-qc9r4"] Jul 06 19:32:42.982568 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:42.982541 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-qc9r4" Jul 06 19:32:42.989812 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:42.989786 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 06 19:32:42.996351 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:42.995934 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-vcvcc\"" Jul 06 19:32:43.021680 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:43.021650 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-qc9r4"] Jul 06 19:32:43.080806 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:43.080771 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bba14ce7-8926-47cd-bf1c-474c7d179cdf-cert\") pod \"odh-model-controller-7998c67dc4-qc9r4\" (UID: \"bba14ce7-8926-47cd-bf1c-474c7d179cdf\") " pod="opendatahub/odh-model-controller-7998c67dc4-qc9r4" Jul 06 19:32:43.080963 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:43.080814 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27ljs\" (UniqueName: \"kubernetes.io/projected/bba14ce7-8926-47cd-bf1c-474c7d179cdf-kube-api-access-27ljs\") pod \"odh-model-controller-7998c67dc4-qc9r4\" (UID: \"bba14ce7-8926-47cd-bf1c-474c7d179cdf\") " pod="opendatahub/odh-model-controller-7998c67dc4-qc9r4" Jul 06 19:32:43.090438 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:43.090399 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c47f5985b-pczqp"] Jul 06 19:32:43.094694 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:32:43.094670 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb77b8d39_d01b_409d_bbbc_1e30d9200e1c.slice/crio-3cc0260370ded7ea2aea557901099140d711582465650d4c6d4927d685f6b35c WatchSource:0}: Error finding container 3cc0260370ded7ea2aea557901099140d711582465650d4c6d4927d685f6b35c: Status 404 returned error can't find the container with id 3cc0260370ded7ea2aea557901099140d711582465650d4c6d4927d685f6b35c Jul 06 19:32:43.182712 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:43.182607 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bba14ce7-8926-47cd-bf1c-474c7d179cdf-cert\") pod \"odh-model-controller-7998c67dc4-qc9r4\" (UID: \"bba14ce7-8926-47cd-bf1c-474c7d179cdf\") " pod="opendatahub/odh-model-controller-7998c67dc4-qc9r4" Jul 06 19:32:43.182883 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:32:43.182739 2575 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 06 19:32:43.182883 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:32:43.182804 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bba14ce7-8926-47cd-bf1c-474c7d179cdf-cert podName:bba14ce7-8926-47cd-bf1c-474c7d179cdf nodeName:}" failed. No retries permitted until 2026-07-06 19:32:43.68278124 +0000 UTC m=+372.502869933 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/bba14ce7-8926-47cd-bf1c-474c7d179cdf-cert") pod "odh-model-controller-7998c67dc4-qc9r4" (UID: "bba14ce7-8926-47cd-bf1c-474c7d179cdf") : secret "odh-model-controller-webhook-cert" not found Jul 06 19:32:43.186207 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:43.183051 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27ljs\" (UniqueName: \"kubernetes.io/projected/bba14ce7-8926-47cd-bf1c-474c7d179cdf-kube-api-access-27ljs\") pod \"odh-model-controller-7998c67dc4-qc9r4\" (UID: \"bba14ce7-8926-47cd-bf1c-474c7d179cdf\") " pod="opendatahub/odh-model-controller-7998c67dc4-qc9r4" Jul 06 19:32:43.199952 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:43.199916 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27ljs\" (UniqueName: \"kubernetes.io/projected/bba14ce7-8926-47cd-bf1c-474c7d179cdf-kube-api-access-27ljs\") pod \"odh-model-controller-7998c67dc4-qc9r4\" (UID: \"bba14ce7-8926-47cd-bf1c-474c7d179cdf\") " pod="opendatahub/odh-model-controller-7998c67dc4-qc9r4" Jul 06 19:32:43.440823 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:43.440735 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c47f5985b-pczqp" event={"ID":"b77b8d39-d01b-409d-bbbc-1e30d9200e1c","Type":"ContainerStarted","Data":"3cc0260370ded7ea2aea557901099140d711582465650d4c6d4927d685f6b35c"} Jul 06 19:32:43.690147 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:43.690106 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bba14ce7-8926-47cd-bf1c-474c7d179cdf-cert\") pod \"odh-model-controller-7998c67dc4-qc9r4\" (UID: \"bba14ce7-8926-47cd-bf1c-474c7d179cdf\") " pod="opendatahub/odh-model-controller-7998c67dc4-qc9r4" Jul 06 19:32:43.692642 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:43.692569 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bba14ce7-8926-47cd-bf1c-474c7d179cdf-cert\") pod \"odh-model-controller-7998c67dc4-qc9r4\" (UID: \"bba14ce7-8926-47cd-bf1c-474c7d179cdf\") " pod="opendatahub/odh-model-controller-7998c67dc4-qc9r4" Jul 06 19:32:43.900983 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:43.900954 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-qc9r4" Jul 06 19:32:44.061876 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:44.061842 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-qc9r4"] Jul 06 19:32:44.064840 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:32:44.064809 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbba14ce7_8926_47cd_bf1c_474c7d179cdf.slice/crio-6e6a77ab52d879bbdff6c85d5880ad0fb441663791c920dbf8990b5aa6487acc WatchSource:0}: Error finding container 6e6a77ab52d879bbdff6c85d5880ad0fb441663791c920dbf8990b5aa6487acc: Status 404 returned error can't find the container with id 6e6a77ab52d879bbdff6c85d5880ad0fb441663791c920dbf8990b5aa6487acc Jul 06 19:32:44.448628 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:44.448594 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-qc9r4" event={"ID":"bba14ce7-8926-47cd-bf1c-474c7d179cdf","Type":"ContainerStarted","Data":"6e6a77ab52d879bbdff6c85d5880ad0fb441663791c920dbf8990b5aa6487acc"} Jul 06 19:32:44.486817 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:44.486783 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-x9n6r"] Jul 06 19:32:44.490308 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:44.490286 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-x9n6r" Jul 06 19:32:44.494277 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:44.494031 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 06 19:32:44.494277 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:44.494068 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-vgpmp\"" Jul 06 19:32:44.499527 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:44.498916 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3e123fa-452d-4cb3-b12e-ff02d37e6ac9-tls-certs\") pod \"model-serving-api-54d89bd5fc-x9n6r\" (UID: \"d3e123fa-452d-4cb3-b12e-ff02d37e6ac9\") " pod="opendatahub/model-serving-api-54d89bd5fc-x9n6r" Jul 06 19:32:44.499527 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:44.498980 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l94bl\" (UniqueName: \"kubernetes.io/projected/d3e123fa-452d-4cb3-b12e-ff02d37e6ac9-kube-api-access-l94bl\") pod \"model-serving-api-54d89bd5fc-x9n6r\" (UID: \"d3e123fa-452d-4cb3-b12e-ff02d37e6ac9\") " pod="opendatahub/model-serving-api-54d89bd5fc-x9n6r" Jul 06 19:32:44.520276 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:44.520241 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-x9n6r"] Jul 06 19:32:44.554949 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:44.554925 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-g9s7w"] Jul 06 19:32:44.557291 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:44.557276 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-g9s7w" Jul 06 19:32:44.560691 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:44.560662 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 06 19:32:44.560810 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:44.560692 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 06 19:32:44.560810 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:44.560672 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 06 19:32:44.560810 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:44.560668 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 06 19:32:44.561296 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:44.561260 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-j9h48\"" Jul 06 19:32:44.600399 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:44.600041 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/74628160-2f6c-422c-ab65-1cf0ddce8e81-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-g9s7w\" (UID: \"74628160-2f6c-422c-ab65-1cf0ddce8e81\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-g9s7w" Jul 06 19:32:44.600399 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:44.600094 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stdvv\" (UniqueName: \"kubernetes.io/projected/74628160-2f6c-422c-ab65-1cf0ddce8e81-kube-api-access-stdvv\") pod \"istiod-openshift-gateway-7574459656-g9s7w\" (UID: \"74628160-2f6c-422c-ab65-1cf0ddce8e81\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-g9s7w" Jul 06 19:32:44.600399 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:44.600173 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/74628160-2f6c-422c-ab65-1cf0ddce8e81-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-g9s7w\" (UID: \"74628160-2f6c-422c-ab65-1cf0ddce8e81\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-g9s7w" Jul 06 19:32:44.600399 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:44.600243 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3e123fa-452d-4cb3-b12e-ff02d37e6ac9-tls-certs\") pod \"model-serving-api-54d89bd5fc-x9n6r\" (UID: \"d3e123fa-452d-4cb3-b12e-ff02d37e6ac9\") " pod="opendatahub/model-serving-api-54d89bd5fc-x9n6r" Jul 06 19:32:44.600399 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:44.600269 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/74628160-2f6c-422c-ab65-1cf0ddce8e81-local-certs\") pod \"istiod-openshift-gateway-7574459656-g9s7w\" (UID: \"74628160-2f6c-422c-ab65-1cf0ddce8e81\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-g9s7w" Jul 06 19:32:44.600399 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:32:44.600376 2575 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 06 19:32:44.600797 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:44.600471 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74628160-2f6c-422c-ab65-1cf0ddce8e81-istio-token\") pod \"istiod-openshift-gateway-7574459656-g9s7w\" (UID: \"74628160-2f6c-422c-ab65-1cf0ddce8e81\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-g9s7w" Jul 06 19:32:44.600797 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:44.600502 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/74628160-2f6c-422c-ab65-1cf0ddce8e81-cacerts\") pod \"istiod-openshift-gateway-7574459656-g9s7w\" (UID: \"74628160-2f6c-422c-ab65-1cf0ddce8e81\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-g9s7w" Jul 06 19:32:44.600797 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:44.600534 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/74628160-2f6c-422c-ab65-1cf0ddce8e81-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-g9s7w\" (UID: \"74628160-2f6c-422c-ab65-1cf0ddce8e81\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-g9s7w" Jul 06 19:32:44.600797 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:32:44.600566 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3e123fa-452d-4cb3-b12e-ff02d37e6ac9-tls-certs podName:d3e123fa-452d-4cb3-b12e-ff02d37e6ac9 nodeName:}" failed. No retries permitted until 2026-07-06 19:32:45.100542463 +0000 UTC m=+373.920631157 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/d3e123fa-452d-4cb3-b12e-ff02d37e6ac9-tls-certs") pod "model-serving-api-54d89bd5fc-x9n6r" (UID: "d3e123fa-452d-4cb3-b12e-ff02d37e6ac9") : secret "model-serving-api-tls" not found Jul 06 19:32:44.600797 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:44.600613 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l94bl\" (UniqueName: \"kubernetes.io/projected/d3e123fa-452d-4cb3-b12e-ff02d37e6ac9-kube-api-access-l94bl\") pod \"model-serving-api-54d89bd5fc-x9n6r\" (UID: \"d3e123fa-452d-4cb3-b12e-ff02d37e6ac9\") " pod="opendatahub/model-serving-api-54d89bd5fc-x9n6r" Jul 06 19:32:44.621175 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:44.621082 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-g9s7w"] Jul 06 19:32:44.633264 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:44.632855 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l94bl\" (UniqueName: \"kubernetes.io/projected/d3e123fa-452d-4cb3-b12e-ff02d37e6ac9-kube-api-access-l94bl\") pod \"model-serving-api-54d89bd5fc-x9n6r\" (UID: \"d3e123fa-452d-4cb3-b12e-ff02d37e6ac9\") " pod="opendatahub/model-serving-api-54d89bd5fc-x9n6r" Jul 06 19:32:44.701477 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:44.701404 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74628160-2f6c-422c-ab65-1cf0ddce8e81-istio-token\") pod \"istiod-openshift-gateway-7574459656-g9s7w\" (UID: \"74628160-2f6c-422c-ab65-1cf0ddce8e81\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-g9s7w" Jul 06 19:32:44.701477 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:44.701448 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/74628160-2f6c-422c-ab65-1cf0ddce8e81-cacerts\") pod \"istiod-openshift-gateway-7574459656-g9s7w\" (UID: \"74628160-2f6c-422c-ab65-1cf0ddce8e81\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-g9s7w" Jul 06 19:32:44.701477 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:44.701476 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/74628160-2f6c-422c-ab65-1cf0ddce8e81-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-g9s7w\" (UID: \"74628160-2f6c-422c-ab65-1cf0ddce8e81\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-g9s7w" Jul 06 19:32:44.701977 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:44.701530 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/74628160-2f6c-422c-ab65-1cf0ddce8e81-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-g9s7w\" (UID: \"74628160-2f6c-422c-ab65-1cf0ddce8e81\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-g9s7w" Jul 06 19:32:44.701977 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:44.701561 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stdvv\" (UniqueName: \"kubernetes.io/projected/74628160-2f6c-422c-ab65-1cf0ddce8e81-kube-api-access-stdvv\") pod \"istiod-openshift-gateway-7574459656-g9s7w\" (UID: \"74628160-2f6c-422c-ab65-1cf0ddce8e81\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-g9s7w" Jul 06 19:32:44.701977 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:44.701626 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/74628160-2f6c-422c-ab65-1cf0ddce8e81-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-g9s7w\" (UID: \"74628160-2f6c-422c-ab65-1cf0ddce8e81\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-g9s7w" Jul 06 19:32:44.701977 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:44.701678 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/74628160-2f6c-422c-ab65-1cf0ddce8e81-local-certs\") pod \"istiod-openshift-gateway-7574459656-g9s7w\" (UID: \"74628160-2f6c-422c-ab65-1cf0ddce8e81\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-g9s7w" Jul 06 19:32:44.703541 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:44.702852 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/74628160-2f6c-422c-ab65-1cf0ddce8e81-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-g9s7w\" (UID: \"74628160-2f6c-422c-ab65-1cf0ddce8e81\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-g9s7w" Jul 06 19:32:44.706723 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:44.706672 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/74628160-2f6c-422c-ab65-1cf0ddce8e81-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-g9s7w\" (UID: \"74628160-2f6c-422c-ab65-1cf0ddce8e81\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-g9s7w" Jul 06 19:32:44.707556 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:44.707528 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/74628160-2f6c-422c-ab65-1cf0ddce8e81-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-g9s7w\" (UID: \"74628160-2f6c-422c-ab65-1cf0ddce8e81\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-g9s7w" Jul 06 19:32:44.707657 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:44.707534 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/74628160-2f6c-422c-ab65-1cf0ddce8e81-cacerts\") pod \"istiod-openshift-gateway-7574459656-g9s7w\" (UID: \"74628160-2f6c-422c-ab65-1cf0ddce8e81\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-g9s7w" Jul 06 19:32:44.709613 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:44.709525 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/74628160-2f6c-422c-ab65-1cf0ddce8e81-local-certs\") pod \"istiod-openshift-gateway-7574459656-g9s7w\" (UID: \"74628160-2f6c-422c-ab65-1cf0ddce8e81\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-g9s7w" Jul 06 19:32:44.716238 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:44.716193 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74628160-2f6c-422c-ab65-1cf0ddce8e81-istio-token\") pod \"istiod-openshift-gateway-7574459656-g9s7w\" (UID: \"74628160-2f6c-422c-ab65-1cf0ddce8e81\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-g9s7w" Jul 06 19:32:44.717276 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:44.717239 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stdvv\" (UniqueName: \"kubernetes.io/projected/74628160-2f6c-422c-ab65-1cf0ddce8e81-kube-api-access-stdvv\") pod \"istiod-openshift-gateway-7574459656-g9s7w\" (UID: \"74628160-2f6c-422c-ab65-1cf0ddce8e81\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-g9s7w" Jul 06 19:32:44.870208 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:44.869879 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-g9s7w" Jul 06 19:32:45.086804 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:45.086169 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-g9s7w"] Jul 06 19:32:45.106217 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:45.105938 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3e123fa-452d-4cb3-b12e-ff02d37e6ac9-tls-certs\") pod \"model-serving-api-54d89bd5fc-x9n6r\" (UID: \"d3e123fa-452d-4cb3-b12e-ff02d37e6ac9\") " pod="opendatahub/model-serving-api-54d89bd5fc-x9n6r" Jul 06 19:32:45.106217 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:32:45.106119 2575 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 06 19:32:45.106798 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:32:45.106452 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3e123fa-452d-4cb3-b12e-ff02d37e6ac9-tls-certs podName:d3e123fa-452d-4cb3-b12e-ff02d37e6ac9 nodeName:}" failed. No retries permitted until 2026-07-06 19:32:46.10616386 +0000 UTC m=+374.926252551 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/d3e123fa-452d-4cb3-b12e-ff02d37e6ac9-tls-certs") pod "model-serving-api-54d89bd5fc-x9n6r" (UID: "d3e123fa-452d-4cb3-b12e-ff02d37e6ac9") : secret "model-serving-api-tls" not found Jul 06 19:32:45.959357 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:32:45.959317 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74628160_2f6c_422c_ab65_1cf0ddce8e81.slice/crio-3750bf1d982b0613aebcbd8830d9dc406d6c2d6ab14acd8005c5904ce9fa9f13 WatchSource:0}: Error finding container 3750bf1d982b0613aebcbd8830d9dc406d6c2d6ab14acd8005c5904ce9fa9f13: Status 404 returned error can't find the container with id 3750bf1d982b0613aebcbd8830d9dc406d6c2d6ab14acd8005c5904ce9fa9f13 Jul 06 19:32:46.116680 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:46.116563 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3e123fa-452d-4cb3-b12e-ff02d37e6ac9-tls-certs\") pod \"model-serving-api-54d89bd5fc-x9n6r\" (UID: \"d3e123fa-452d-4cb3-b12e-ff02d37e6ac9\") " pod="opendatahub/model-serving-api-54d89bd5fc-x9n6r" Jul 06 19:32:46.124970 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:46.124939 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3e123fa-452d-4cb3-b12e-ff02d37e6ac9-tls-certs\") pod \"model-serving-api-54d89bd5fc-x9n6r\" (UID: \"d3e123fa-452d-4cb3-b12e-ff02d37e6ac9\") " pod="opendatahub/model-serving-api-54d89bd5fc-x9n6r" Jul 06 19:32:46.303421 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:46.303341 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-x9n6r" Jul 06 19:32:46.462953 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:46.462917 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-g9s7w" event={"ID":"74628160-2f6c-422c-ab65-1cf0ddce8e81","Type":"ContainerStarted","Data":"3750bf1d982b0613aebcbd8830d9dc406d6c2d6ab14acd8005c5904ce9fa9f13"} Jul 06 19:32:47.479309 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:47.478457 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c47f5985b-pczqp" event={"ID":"b77b8d39-d01b-409d-bbbc-1e30d9200e1c","Type":"ContainerStarted","Data":"985472e0f534f07d4e7a6a5444d198b975114bd116d2f03703eb4e7edec83505"} Jul 06 19:32:47.484341 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:47.484310 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-qc9r4" event={"ID":"bba14ce7-8926-47cd-bf1c-474c7d179cdf","Type":"ContainerStarted","Data":"798211a939e8f4b52490eee5b99ab340b10db5f1028af2a9921ab5439f36076e"} Jul 06 19:32:47.485162 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:47.485102 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-qc9r4" Jul 06 19:32:47.510077 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:47.507492 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7c47f5985b-pczqp" podStartSLOduration=1.247267847 podStartE2EDuration="5.507470255s" totalImagesPullingTime="4.260202408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:32:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:32:47.506440017 +0000 UTC m=+376.326528743" watchObservedRunningTime="2026-07-06 19:32:47.507470255 +0000 UTC m=+376.327558966" Jul 06 19:32:47.518863 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:47.516519 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-x9n6r"] Jul 06 19:32:47.538252 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:47.538198 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-qc9r4" podStartSLOduration=2.234657365 podStartE2EDuration="5.53816645s" totalImagesPullingTime="3.303509085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:32:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:32:47.536963151 +0000 UTC m=+376.357051875" watchObservedRunningTime="2026-07-06 19:32:47.53816645 +0000 UTC m=+376.358255162" Jul 06 19:32:48.492156 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:48.492120 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-x9n6r" event={"ID":"d3e123fa-452d-4cb3-b12e-ff02d37e6ac9","Type":"ContainerStarted","Data":"71eca0f8bf1b3343afd70e3e70edad408182190384b13201e22cda4fa4d73820"} Jul 06 19:32:48.740785 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:48.740746 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 19:32:48.740875 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:48.740820 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 19:32:49.501670 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:49.501589 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-g9s7w" event={"ID":"74628160-2f6c-422c-ab65-1cf0ddce8e81","Type":"ContainerStarted","Data":"eb83757d87861ca104346e71bc64792914f9199e59a5d70ff74f20683d113bd3"} Jul 06 19:32:49.502175 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:49.501880 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-g9s7w" Jul 06 19:32:49.503688 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:49.503663 2575 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-g9s7w container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 19:32:49.503791 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:49.503731 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-g9s7w" podUID="74628160-2f6c-422c-ab65-1cf0ddce8e81" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 19:32:49.536370 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:49.536327 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-g9s7w" podStartSLOduration=2.758668278 podStartE2EDuration="5.536311452s" totalImagesPullingTime="2.777643174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:32:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:32:49.532930904 +0000 UTC m=+378.353019617" watchObservedRunningTime="2026-07-06 19:32:49.536311452 +0000 UTC m=+378.356400161" Jul 06 19:32:50.510633 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:50.510435 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-x9n6r" event={"ID":"d3e123fa-452d-4cb3-b12e-ff02d37e6ac9","Type":"ContainerStarted","Data":"771f537911ceae597d2c920375b62f31736d5059160750ec2c75f25a76c575ac"} Jul 06 19:32:50.510633 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:50.510541 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-x9n6r" Jul 06 19:32:50.511575 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:50.511550 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-g9s7w" Jul 06 19:32:50.577083 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:50.577040 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-x9n6r" podStartSLOduration=3.949335896 podStartE2EDuration="6.577022362s" totalImagesPullingTime="2.627686466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:32:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:32:50.57578639 +0000 UTC m=+379.395875101" watchObservedRunningTime="2026-07-06 19:32:50.577022362 +0000 UTC m=+379.397111071" Jul 06 19:32:55.323837 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:55.323798 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-l8hsx"] Jul 06 19:32:55.326446 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:55.326427 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-l8hsx" Jul 06 19:32:55.330031 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:55.330009 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 06 19:32:55.330938 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:55.330923 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-mlznh\"" Jul 06 19:32:55.344830 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:55.344805 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-8vh5h"] Jul 06 19:32:55.347875 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:55.347856 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8vh5h" Jul 06 19:32:55.352455 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:55.352430 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 06 19:32:55.352841 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:55.352818 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-99kmk\"" Jul 06 19:32:55.353073 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:55.353041 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-l8hsx"] Jul 06 19:32:55.377481 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:55.377454 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-8vh5h"] Jul 06 19:32:55.402927 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:55.402905 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c159251-98a2-4993-af7d-aecf281fbf67-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-8vh5h\" (UID: \"1c159251-98a2-4993-af7d-aecf281fbf67\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8vh5h" Jul 06 19:32:55.403062 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:55.402961 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/67cd03db-d916-4703-9f42-b5e0df241829-cert\") pod \"kserve-controller-manager-d6f4b65b8-l8hsx\" (UID: \"67cd03db-d916-4703-9f42-b5e0df241829\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-l8hsx" Jul 06 19:32:55.403062 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:55.403017 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xn9m\" (UniqueName: \"kubernetes.io/projected/1c159251-98a2-4993-af7d-aecf281fbf67-kube-api-access-9xn9m\") pod \"llmisvc-controller-manager-777c6bd7d5-8vh5h\" (UID: \"1c159251-98a2-4993-af7d-aecf281fbf67\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8vh5h" Jul 06 19:32:55.403170 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:55.403064 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzl6q\" (UniqueName: \"kubernetes.io/projected/67cd03db-d916-4703-9f42-b5e0df241829-kube-api-access-gzl6q\") pod \"kserve-controller-manager-d6f4b65b8-l8hsx\" (UID: \"67cd03db-d916-4703-9f42-b5e0df241829\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-l8hsx" Jul 06 19:32:55.503689 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:55.503663 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/67cd03db-d916-4703-9f42-b5e0df241829-cert\") pod \"kserve-controller-manager-d6f4b65b8-l8hsx\" (UID: \"67cd03db-d916-4703-9f42-b5e0df241829\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-l8hsx" Jul 06 19:32:55.503849 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:55.503713 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xn9m\" (UniqueName: \"kubernetes.io/projected/1c159251-98a2-4993-af7d-aecf281fbf67-kube-api-access-9xn9m\") pod \"llmisvc-controller-manager-777c6bd7d5-8vh5h\" (UID: \"1c159251-98a2-4993-af7d-aecf281fbf67\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8vh5h" Jul 06 19:32:55.503849 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:55.503744 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzl6q\" (UniqueName: \"kubernetes.io/projected/67cd03db-d916-4703-9f42-b5e0df241829-kube-api-access-gzl6q\") pod \"kserve-controller-manager-d6f4b65b8-l8hsx\" (UID: \"67cd03db-d916-4703-9f42-b5e0df241829\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-l8hsx" Jul 06 19:32:55.503849 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:55.503762 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c159251-98a2-4993-af7d-aecf281fbf67-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-8vh5h\" (UID: \"1c159251-98a2-4993-af7d-aecf281fbf67\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8vh5h" Jul 06 19:32:55.503849 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:32:55.503842 2575 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 06 19:32:55.504046 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:32:55.503892 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c159251-98a2-4993-af7d-aecf281fbf67-cert podName:1c159251-98a2-4993-af7d-aecf281fbf67 nodeName:}" failed. No retries permitted until 2026-07-06 19:32:56.003874368 +0000 UTC m=+384.823963057 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1c159251-98a2-4993-af7d-aecf281fbf67-cert") pod "llmisvc-controller-manager-777c6bd7d5-8vh5h" (UID: "1c159251-98a2-4993-af7d-aecf281fbf67") : secret "llmisvc-webhook-server-cert" not found Jul 06 19:32:55.506252 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:55.506224 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/67cd03db-d916-4703-9f42-b5e0df241829-cert\") pod \"kserve-controller-manager-d6f4b65b8-l8hsx\" (UID: \"67cd03db-d916-4703-9f42-b5e0df241829\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-l8hsx" Jul 06 19:32:55.527809 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:55.527786 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzl6q\" (UniqueName: \"kubernetes.io/projected/67cd03db-d916-4703-9f42-b5e0df241829-kube-api-access-gzl6q\") pod \"kserve-controller-manager-d6f4b65b8-l8hsx\" (UID: \"67cd03db-d916-4703-9f42-b5e0df241829\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-l8hsx" Jul 06 19:32:55.532962 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:55.532944 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xn9m\" (UniqueName: \"kubernetes.io/projected/1c159251-98a2-4993-af7d-aecf281fbf67-kube-api-access-9xn9m\") pod \"llmisvc-controller-manager-777c6bd7d5-8vh5h\" (UID: \"1c159251-98a2-4993-af7d-aecf281fbf67\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8vh5h" Jul 06 19:32:55.637903 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:55.637842 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-l8hsx" Jul 06 19:32:55.805507 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:55.805484 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-l8hsx"] Jul 06 19:32:55.807201 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:32:55.807154 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67cd03db_d916_4703_9f42_b5e0df241829.slice/crio-8de0ca4b3a3ac95264f6cc184b8f5cc514127ed4db400c30ddeb242e0c89282a WatchSource:0}: Error finding container 8de0ca4b3a3ac95264f6cc184b8f5cc514127ed4db400c30ddeb242e0c89282a: Status 404 returned error can't find the container with id 8de0ca4b3a3ac95264f6cc184b8f5cc514127ed4db400c30ddeb242e0c89282a Jul 06 19:32:56.010028 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:56.009954 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c159251-98a2-4993-af7d-aecf281fbf67-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-8vh5h\" (UID: \"1c159251-98a2-4993-af7d-aecf281fbf67\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8vh5h" Jul 06 19:32:56.012400 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:56.012368 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c159251-98a2-4993-af7d-aecf281fbf67-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-8vh5h\" (UID: \"1c159251-98a2-4993-af7d-aecf281fbf67\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8vh5h" Jul 06 19:32:56.261371 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:56.261296 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8vh5h" Jul 06 19:32:56.427092 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:56.427065 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-8vh5h"] Jul 06 19:32:56.429125 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:32:56.429084 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1c159251_98a2_4993_af7d_aecf281fbf67.slice/crio-aaebb86f27f3335d56f90ce9629937ea5401f567a2588f1ce517132651177ad8 WatchSource:0}: Error finding container aaebb86f27f3335d56f90ce9629937ea5401f567a2588f1ce517132651177ad8: Status 404 returned error can't find the container with id aaebb86f27f3335d56f90ce9629937ea5401f567a2588f1ce517132651177ad8 Jul 06 19:32:56.541362 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:56.541304 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8vh5h" event={"ID":"1c159251-98a2-4993-af7d-aecf281fbf67","Type":"ContainerStarted","Data":"aaebb86f27f3335d56f90ce9629937ea5401f567a2588f1ce517132651177ad8"} Jul 06 19:32:56.542484 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:56.542462 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-l8hsx" event={"ID":"67cd03db-d916-4703-9f42-b5e0df241829","Type":"ContainerStarted","Data":"8de0ca4b3a3ac95264f6cc184b8f5cc514127ed4db400c30ddeb242e0c89282a"} Jul 06 19:32:57.926892 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:57.926856 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-g9s7w"] Jul 06 19:32:57.927494 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:57.927082 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-g9s7w" podUID="74628160-2f6c-422c-ab65-1cf0ddce8e81" containerName="discovery" containerID="cri-o://eb83757d87861ca104346e71bc64792914f9199e59a5d70ff74f20683d113bd3" gracePeriod=30 Jul 06 19:32:58.422267 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:58.422241 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-g9s7w" Jul 06 19:32:58.494771 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:58.494747 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-qc9r4" Jul 06 19:32:58.536815 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:58.536742 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74628160-2f6c-422c-ab65-1cf0ddce8e81-istio-token\") pod \"74628160-2f6c-422c-ab65-1cf0ddce8e81\" (UID: \"74628160-2f6c-422c-ab65-1cf0ddce8e81\") " Jul 06 19:32:58.536815 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:58.536806 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-stdvv\" (UniqueName: \"kubernetes.io/projected/74628160-2f6c-422c-ab65-1cf0ddce8e81-kube-api-access-stdvv\") pod \"74628160-2f6c-422c-ab65-1cf0ddce8e81\" (UID: \"74628160-2f6c-422c-ab65-1cf0ddce8e81\") " Jul 06 19:32:58.537064 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:58.536837 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/74628160-2f6c-422c-ab65-1cf0ddce8e81-istio-csr-dns-cert\") pod \"74628160-2f6c-422c-ab65-1cf0ddce8e81\" (UID: \"74628160-2f6c-422c-ab65-1cf0ddce8e81\") " Jul 06 19:32:58.537064 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:58.536951 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/74628160-2f6c-422c-ab65-1cf0ddce8e81-local-certs\") pod \"74628160-2f6c-422c-ab65-1cf0ddce8e81\" (UID: \"74628160-2f6c-422c-ab65-1cf0ddce8e81\") " Jul 06 19:32:58.537064 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:58.536985 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/74628160-2f6c-422c-ab65-1cf0ddce8e81-istio-csr-ca-configmap\") pod \"74628160-2f6c-422c-ab65-1cf0ddce8e81\" (UID: \"74628160-2f6c-422c-ab65-1cf0ddce8e81\") " Jul 06 19:32:58.537064 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:58.537033 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/74628160-2f6c-422c-ab65-1cf0ddce8e81-istio-kubeconfig\") pod \"74628160-2f6c-422c-ab65-1cf0ddce8e81\" (UID: \"74628160-2f6c-422c-ab65-1cf0ddce8e81\") " Jul 06 19:32:58.537064 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:58.537060 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/74628160-2f6c-422c-ab65-1cf0ddce8e81-cacerts\") pod \"74628160-2f6c-422c-ab65-1cf0ddce8e81\" (UID: \"74628160-2f6c-422c-ab65-1cf0ddce8e81\") " Jul 06 19:32:58.538437 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:58.538023 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74628160-2f6c-422c-ab65-1cf0ddce8e81-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "74628160-2f6c-422c-ab65-1cf0ddce8e81" (UID: "74628160-2f6c-422c-ab65-1cf0ddce8e81"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:32:58.543075 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:58.543017 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74628160-2f6c-422c-ab65-1cf0ddce8e81-cacerts" (OuterVolumeSpecName: "cacerts") pod "74628160-2f6c-422c-ab65-1cf0ddce8e81" (UID: "74628160-2f6c-422c-ab65-1cf0ddce8e81"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:32:58.545214 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:58.545161 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74628160-2f6c-422c-ab65-1cf0ddce8e81-kube-api-access-stdvv" (OuterVolumeSpecName: "kube-api-access-stdvv") pod "74628160-2f6c-422c-ab65-1cf0ddce8e81" (UID: "74628160-2f6c-422c-ab65-1cf0ddce8e81"). InnerVolumeSpecName "kube-api-access-stdvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:32:58.548644 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:58.548621 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74628160-2f6c-422c-ab65-1cf0ddce8e81-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "74628160-2f6c-422c-ab65-1cf0ddce8e81" (UID: "74628160-2f6c-422c-ab65-1cf0ddce8e81"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:32:58.549080 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:58.549041 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74628160-2f6c-422c-ab65-1cf0ddce8e81-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "74628160-2f6c-422c-ab65-1cf0ddce8e81" (UID: "74628160-2f6c-422c-ab65-1cf0ddce8e81"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:32:58.549609 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:58.549370 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74628160-2f6c-422c-ab65-1cf0ddce8e81-local-certs" (OuterVolumeSpecName: "local-certs") pod "74628160-2f6c-422c-ab65-1cf0ddce8e81" (UID: "74628160-2f6c-422c-ab65-1cf0ddce8e81"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:32:58.549609 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:58.549552 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74628160-2f6c-422c-ab65-1cf0ddce8e81-istio-token" (OuterVolumeSpecName: "istio-token") pod "74628160-2f6c-422c-ab65-1cf0ddce8e81" (UID: "74628160-2f6c-422c-ab65-1cf0ddce8e81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:32:58.560402 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:58.560366 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-l8hsx" event={"ID":"67cd03db-d916-4703-9f42-b5e0df241829","Type":"ContainerStarted","Data":"ce6539d177f1f11c19f527195e4576218a95a3ce9e5647e186c0590c4d312081"} Jul 06 19:32:58.560506 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:58.560460 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-l8hsx" Jul 06 19:32:58.562888 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:58.562865 2575 generic.go:356] "Generic (PLEG): container finished" podID="74628160-2f6c-422c-ab65-1cf0ddce8e81" containerID="eb83757d87861ca104346e71bc64792914f9199e59a5d70ff74f20683d113bd3" exitCode=0 Jul 06 19:32:58.563034 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:58.563009 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-g9s7w" Jul 06 19:32:58.563034 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:58.563018 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-g9s7w" event={"ID":"74628160-2f6c-422c-ab65-1cf0ddce8e81","Type":"ContainerDied","Data":"eb83757d87861ca104346e71bc64792914f9199e59a5d70ff74f20683d113bd3"} Jul 06 19:32:58.563225 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:58.563074 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-g9s7w" event={"ID":"74628160-2f6c-422c-ab65-1cf0ddce8e81","Type":"ContainerDied","Data":"3750bf1d982b0613aebcbd8830d9dc406d6c2d6ab14acd8005c5904ce9fa9f13"} Jul 06 19:32:58.563225 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:58.563104 2575 scope.go:117] "RemoveContainer" containerID="eb83757d87861ca104346e71bc64792914f9199e59a5d70ff74f20683d113bd3" Jul 06 19:32:58.574831 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:58.574811 2575 scope.go:117] "RemoveContainer" containerID="eb83757d87861ca104346e71bc64792914f9199e59a5d70ff74f20683d113bd3" Jul 06 19:32:58.575170 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:32:58.575141 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb83757d87861ca104346e71bc64792914f9199e59a5d70ff74f20683d113bd3\": container with ID starting with eb83757d87861ca104346e71bc64792914f9199e59a5d70ff74f20683d113bd3 not found: ID does not exist" containerID="eb83757d87861ca104346e71bc64792914f9199e59a5d70ff74f20683d113bd3" Jul 06 19:32:58.575280 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:58.575200 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb83757d87861ca104346e71bc64792914f9199e59a5d70ff74f20683d113bd3"} err="failed to get container status \"eb83757d87861ca104346e71bc64792914f9199e59a5d70ff74f20683d113bd3\": rpc error: code = NotFound desc = could not find container \"eb83757d87861ca104346e71bc64792914f9199e59a5d70ff74f20683d113bd3\": container with ID starting with eb83757d87861ca104346e71bc64792914f9199e59a5d70ff74f20683d113bd3 not found: ID does not exist" Jul 06 19:32:58.606291 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:58.606155 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-l8hsx" podStartSLOduration=1.133720801 podStartE2EDuration="3.606135543s" totalImagesPullingTime="2.472414742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:32:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:32:58.584940876 +0000 UTC m=+387.405029588" watchObservedRunningTime="2026-07-06 19:32:58.606135543 +0000 UTC m=+387.426224256" Jul 06 19:32:58.606648 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:58.606621 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-g9s7w"] Jul 06 19:32:58.608637 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:58.608619 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-g9s7w"] Jul 06 19:32:58.638156 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:58.638135 2575 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/74628160-2f6c-422c-ab65-1cf0ddce8e81-istio-kubeconfig\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:32:58.638156 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:58.638153 2575 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/74628160-2f6c-422c-ab65-1cf0ddce8e81-cacerts\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:32:58.638331 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:58.638165 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74628160-2f6c-422c-ab65-1cf0ddce8e81-istio-token\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:32:58.638331 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:58.638177 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stdvv\" (UniqueName: \"kubernetes.io/projected/74628160-2f6c-422c-ab65-1cf0ddce8e81-kube-api-access-stdvv\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:32:58.638331 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:58.638210 2575 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/74628160-2f6c-422c-ab65-1cf0ddce8e81-istio-csr-dns-cert\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:32:58.638331 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:58.638219 2575 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/74628160-2f6c-422c-ab65-1cf0ddce8e81-local-certs\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:32:58.638331 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:58.638228 2575 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/74628160-2f6c-422c-ab65-1cf0ddce8e81-istio-csr-ca-configmap\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:32:59.571100 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:59.571049 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8vh5h" event={"ID":"1c159251-98a2-4993-af7d-aecf281fbf67","Type":"ContainerStarted","Data":"8e85871941e3b1613d6a2d7677f946e788137fd3f1e2f99a6fc1d20117458f49"} Jul 06 19:32:59.571455 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:59.571221 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8vh5h" Jul 06 19:32:59.594472 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:59.594423 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8vh5h" podStartSLOduration=1.6551729389999998 podStartE2EDuration="4.594408594s" totalImagesPullingTime="2.939235655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:32:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:32:59.592488192 +0000 UTC m=+388.412576906" watchObservedRunningTime="2026-07-06 19:32:59.594408594 +0000 UTC m=+388.414497304" Jul 06 19:32:59.797466 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:32:59.797427 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74628160-2f6c-422c-ab65-1cf0ddce8e81" path="/var/lib/kubelet/pods/74628160-2f6c-422c-ab65-1cf0ddce8e81/volumes" Jul 06 19:33:01.518462 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:01.518436 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-x9n6r" Jul 06 19:33:29.577462 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:29.577429 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-l8hsx" Jul 06 19:33:30.579076 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:30.579049 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8vh5h" Jul 06 19:33:44.470819 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:44.470782 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gchlg"] Jul 06 19:33:44.471349 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:44.471326 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74628160-2f6c-422c-ab65-1cf0ddce8e81" containerName="discovery" Jul 06 19:33:44.471349 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:44.471345 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="74628160-2f6c-422c-ab65-1cf0ddce8e81" containerName="discovery" Jul 06 19:33:44.471552 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:44.471433 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="74628160-2f6c-422c-ab65-1cf0ddce8e81" containerName="discovery" Jul 06 19:33:44.475884 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:44.475861 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gchlg" Jul 06 19:33:44.478848 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:44.478818 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 19:33:44.478848 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:44.478835 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-kgdf9\"" Jul 06 19:33:44.479810 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:44.479792 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 19:33:44.486016 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:44.485962 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gchlg"] Jul 06 19:33:44.493602 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:44.493578 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tzb7\" (UniqueName: \"kubernetes.io/projected/c7f911fe-1194-46f3-936a-6adf2f308a7a-kube-api-access-5tzb7\") pod \"kuadrant-operator-catalog-gchlg\" (UID: \"c7f911fe-1194-46f3-936a-6adf2f308a7a\") " pod="kuadrant-system/kuadrant-operator-catalog-gchlg" Jul 06 19:33:44.594553 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:44.594532 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tzb7\" (UniqueName: \"kubernetes.io/projected/c7f911fe-1194-46f3-936a-6adf2f308a7a-kube-api-access-5tzb7\") pod \"kuadrant-operator-catalog-gchlg\" (UID: \"c7f911fe-1194-46f3-936a-6adf2f308a7a\") " pod="kuadrant-system/kuadrant-operator-catalog-gchlg" Jul 06 19:33:44.603588 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:44.603566 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tzb7\" (UniqueName: \"kubernetes.io/projected/c7f911fe-1194-46f3-936a-6adf2f308a7a-kube-api-access-5tzb7\") pod \"kuadrant-operator-catalog-gchlg\" (UID: \"c7f911fe-1194-46f3-936a-6adf2f308a7a\") " pod="kuadrant-system/kuadrant-operator-catalog-gchlg" Jul 06 19:33:44.787051 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:44.786985 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gchlg" Jul 06 19:33:44.802486 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:44.802460 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gchlg"] Jul 06 19:33:44.922548 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:44.922525 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gchlg"] Jul 06 19:33:44.924449 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:33:44.924426 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7f911fe_1194_46f3_936a_6adf2f308a7a.slice/crio-2054080ad6d8cdca43553691032e06bcab49def1f102d29798e04ad4d35e46f5 WatchSource:0}: Error finding container 2054080ad6d8cdca43553691032e06bcab49def1f102d29798e04ad4d35e46f5: Status 404 returned error can't find the container with id 2054080ad6d8cdca43553691032e06bcab49def1f102d29798e04ad4d35e46f5 Jul 06 19:33:45.023657 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:45.023637 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-f2kc9"] Jul 06 19:33:45.026464 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:45.026450 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-f2kc9" Jul 06 19:33:45.051300 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:45.051252 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-f2kc9"] Jul 06 19:33:45.099488 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:45.099466 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh5v6\" (UniqueName: \"kubernetes.io/projected/7cafd8f9-3644-4b3a-a810-de3ebf82d7fd-kube-api-access-dh5v6\") pod \"kuadrant-operator-catalog-f2kc9\" (UID: \"7cafd8f9-3644-4b3a-a810-de3ebf82d7fd\") " pod="kuadrant-system/kuadrant-operator-catalog-f2kc9" Jul 06 19:33:45.199975 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:45.199950 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh5v6\" (UniqueName: \"kubernetes.io/projected/7cafd8f9-3644-4b3a-a810-de3ebf82d7fd-kube-api-access-dh5v6\") pod \"kuadrant-operator-catalog-f2kc9\" (UID: \"7cafd8f9-3644-4b3a-a810-de3ebf82d7fd\") " pod="kuadrant-system/kuadrant-operator-catalog-f2kc9" Jul 06 19:33:45.221409 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:45.221385 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh5v6\" (UniqueName: \"kubernetes.io/projected/7cafd8f9-3644-4b3a-a810-de3ebf82d7fd-kube-api-access-dh5v6\") pod \"kuadrant-operator-catalog-f2kc9\" (UID: \"7cafd8f9-3644-4b3a-a810-de3ebf82d7fd\") " pod="kuadrant-system/kuadrant-operator-catalog-f2kc9" Jul 06 19:33:45.335087 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:45.335067 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-f2kc9" Jul 06 19:33:45.480716 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:45.480693 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-f2kc9"] Jul 06 19:33:45.553083 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:33:45.553050 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cafd8f9_3644_4b3a_a810_de3ebf82d7fd.slice/crio-816506532f0af194116241f0da7d108237eebaf79d6a9878e2c464ced7232dc6 WatchSource:0}: Error finding container 816506532f0af194116241f0da7d108237eebaf79d6a9878e2c464ced7232dc6: Status 404 returned error can't find the container with id 816506532f0af194116241f0da7d108237eebaf79d6a9878e2c464ced7232dc6 Jul 06 19:33:45.814227 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:45.814103 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-f2kc9" event={"ID":"7cafd8f9-3644-4b3a-a810-de3ebf82d7fd","Type":"ContainerStarted","Data":"816506532f0af194116241f0da7d108237eebaf79d6a9878e2c464ced7232dc6"} Jul 06 19:33:45.815786 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:45.815733 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gchlg" event={"ID":"c7f911fe-1194-46f3-936a-6adf2f308a7a","Type":"ContainerStarted","Data":"2054080ad6d8cdca43553691032e06bcab49def1f102d29798e04ad4d35e46f5"} Jul 06 19:33:46.825682 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:46.825645 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gchlg" event={"ID":"c7f911fe-1194-46f3-936a-6adf2f308a7a","Type":"ContainerStarted","Data":"e3b2277427e9f307c66824bd630766ec440c0328b432f9829e2a23845aa8e1f5"} Jul 06 19:33:46.826062 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:46.825733 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-gchlg" podUID="c7f911fe-1194-46f3-936a-6adf2f308a7a" containerName="registry-server" containerID="cri-o://e3b2277427e9f307c66824bd630766ec440c0328b432f9829e2a23845aa8e1f5" gracePeriod=2 Jul 06 19:33:46.827414 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:46.827368 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-f2kc9" event={"ID":"7cafd8f9-3644-4b3a-a810-de3ebf82d7fd","Type":"ContainerStarted","Data":"af3718756fdd0d0f44e2e8bd0508f7f0699093043c7c1ca84992f1041b2cb909"} Jul 06 19:33:46.845297 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:46.845253 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-gchlg" podStartSLOduration=1.049197937 podStartE2EDuration="2.845239236s" totalImagesPullingTime="1.796041299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:33:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:33:46.843538518 +0000 UTC m=+435.663627231" watchObservedRunningTime="2026-07-06 19:33:46.845239236 +0000 UTC m=+435.665327981" Jul 06 19:33:46.866577 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:46.866529 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-f2kc9" podStartSLOduration=1.6985008129999999 podStartE2EDuration="2.866515258s" totalImagesPullingTime="1.168014445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:33:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:33:46.8659778 +0000 UTC m=+435.686066502" watchObservedRunningTime="2026-07-06 19:33:46.866515258 +0000 UTC m=+435.686603968" Jul 06 19:33:47.065042 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:47.065019 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-gchlg_c7f911fe-1194-46f3-936a-6adf2f308a7a/registry-server/0.log" Jul 06 19:33:47.067343 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:47.067329 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gchlg" Jul 06 19:33:47.118631 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:47.118606 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tzb7\" (UniqueName: \"kubernetes.io/projected/c7f911fe-1194-46f3-936a-6adf2f308a7a-kube-api-access-5tzb7\") pod \"c7f911fe-1194-46f3-936a-6adf2f308a7a\" (UID: \"c7f911fe-1194-46f3-936a-6adf2f308a7a\") " Jul 06 19:33:47.120733 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:47.120712 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7f911fe-1194-46f3-936a-6adf2f308a7a-kube-api-access-5tzb7" (OuterVolumeSpecName: "kube-api-access-5tzb7") pod "c7f911fe-1194-46f3-936a-6adf2f308a7a" (UID: "c7f911fe-1194-46f3-936a-6adf2f308a7a"). InnerVolumeSpecName "kube-api-access-5tzb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:33:47.219917 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:47.219892 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tzb7\" (UniqueName: \"kubernetes.io/projected/c7f911fe-1194-46f3-936a-6adf2f308a7a-kube-api-access-5tzb7\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:33:47.833824 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:47.833805 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-gchlg_c7f911fe-1194-46f3-936a-6adf2f308a7a/registry-server/0.log" Jul 06 19:33:47.834147 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:47.833838 2575 generic.go:356] "Generic (PLEG): container finished" podID="c7f911fe-1194-46f3-936a-6adf2f308a7a" containerID="e3b2277427e9f307c66824bd630766ec440c0328b432f9829e2a23845aa8e1f5" exitCode=1 Jul 06 19:33:47.834147 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:47.833883 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gchlg" Jul 06 19:33:47.834147 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:47.833923 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gchlg" event={"ID":"c7f911fe-1194-46f3-936a-6adf2f308a7a","Type":"ContainerDied","Data":"e3b2277427e9f307c66824bd630766ec440c0328b432f9829e2a23845aa8e1f5"} Jul 06 19:33:47.834147 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:47.833963 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gchlg" event={"ID":"c7f911fe-1194-46f3-936a-6adf2f308a7a","Type":"ContainerDied","Data":"2054080ad6d8cdca43553691032e06bcab49def1f102d29798e04ad4d35e46f5"} Jul 06 19:33:47.834147 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:47.833988 2575 scope.go:117] "RemoveContainer" containerID="e3b2277427e9f307c66824bd630766ec440c0328b432f9829e2a23845aa8e1f5" Jul 06 19:33:47.849404 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:47.849381 2575 scope.go:117] "RemoveContainer" containerID="e3b2277427e9f307c66824bd630766ec440c0328b432f9829e2a23845aa8e1f5" Jul 06 19:33:47.849662 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:33:47.849641 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3b2277427e9f307c66824bd630766ec440c0328b432f9829e2a23845aa8e1f5\": container with ID starting with e3b2277427e9f307c66824bd630766ec440c0328b432f9829e2a23845aa8e1f5 not found: ID does not exist" containerID="e3b2277427e9f307c66824bd630766ec440c0328b432f9829e2a23845aa8e1f5" Jul 06 19:33:47.849715 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:47.849670 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3b2277427e9f307c66824bd630766ec440c0328b432f9829e2a23845aa8e1f5"} err="failed to get container status \"e3b2277427e9f307c66824bd630766ec440c0328b432f9829e2a23845aa8e1f5\": rpc error: code = NotFound desc = could not find container \"e3b2277427e9f307c66824bd630766ec440c0328b432f9829e2a23845aa8e1f5\": container with ID starting with e3b2277427e9f307c66824bd630766ec440c0328b432f9829e2a23845aa8e1f5 not found: ID does not exist" Jul 06 19:33:47.859498 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:47.859478 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gchlg"] Jul 06 19:33:47.866686 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:47.866663 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gchlg"] Jul 06 19:33:49.794544 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:49.794512 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7f911fe-1194-46f3-936a-6adf2f308a7a" path="/var/lib/kubelet/pods/c7f911fe-1194-46f3-936a-6adf2f308a7a/volumes" Jul 06 19:33:55.335848 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:55.335770 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-f2kc9" Jul 06 19:33:55.336252 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:55.335914 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-f2kc9" Jul 06 19:33:55.357951 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:55.357927 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-f2kc9" Jul 06 19:33:55.896636 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:33:55.896613 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-f2kc9" Jul 06 19:34:00.065081 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:00.065049 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dbs8c"] Jul 06 19:34:00.065457 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:00.065444 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7f911fe-1194-46f3-936a-6adf2f308a7a" containerName="registry-server" Jul 06 19:34:00.065457 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:00.065456 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7f911fe-1194-46f3-936a-6adf2f308a7a" containerName="registry-server" Jul 06 19:34:00.065550 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:00.065539 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7f911fe-1194-46f3-936a-6adf2f308a7a" containerName="registry-server" Jul 06 19:34:00.069234 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:00.069218 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dbs8c" Jul 06 19:34:00.071843 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:00.071819 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-62dw5\"" Jul 06 19:34:00.079550 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:00.079528 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dbs8c"] Jul 06 19:34:00.111827 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:00.111808 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb57645a-f2c3-4241-8929-2e058bfbffbe-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dbs8c\" (UID: \"cb57645a-f2c3-4241-8929-2e058bfbffbe\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dbs8c" Jul 06 19:34:00.111923 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:00.111845 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5fvq\" (UniqueName: \"kubernetes.io/projected/cb57645a-f2c3-4241-8929-2e058bfbffbe-kube-api-access-x5fvq\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dbs8c\" (UID: \"cb57645a-f2c3-4241-8929-2e058bfbffbe\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dbs8c" Jul 06 19:34:00.111923 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:00.111863 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb57645a-f2c3-4241-8929-2e058bfbffbe-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dbs8c\" (UID: \"cb57645a-f2c3-4241-8929-2e058bfbffbe\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dbs8c" Jul 06 19:34:00.212833 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:00.212808 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5fvq\" (UniqueName: \"kubernetes.io/projected/cb57645a-f2c3-4241-8929-2e058bfbffbe-kube-api-access-x5fvq\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dbs8c\" (UID: \"cb57645a-f2c3-4241-8929-2e058bfbffbe\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dbs8c" Jul 06 19:34:00.212930 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:00.212839 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb57645a-f2c3-4241-8929-2e058bfbffbe-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dbs8c\" (UID: \"cb57645a-f2c3-4241-8929-2e058bfbffbe\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dbs8c" Jul 06 19:34:00.212930 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:00.212910 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb57645a-f2c3-4241-8929-2e058bfbffbe-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dbs8c\" (UID: \"cb57645a-f2c3-4241-8929-2e058bfbffbe\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dbs8c" Jul 06 19:34:00.213220 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:00.213205 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb57645a-f2c3-4241-8929-2e058bfbffbe-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dbs8c\" (UID: \"cb57645a-f2c3-4241-8929-2e058bfbffbe\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dbs8c" Jul 06 19:34:00.213273 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:00.213258 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb57645a-f2c3-4241-8929-2e058bfbffbe-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dbs8c\" (UID: \"cb57645a-f2c3-4241-8929-2e058bfbffbe\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dbs8c" Jul 06 19:34:00.224066 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:00.224047 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5fvq\" (UniqueName: \"kubernetes.io/projected/cb57645a-f2c3-4241-8929-2e058bfbffbe-kube-api-access-x5fvq\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dbs8c\" (UID: \"cb57645a-f2c3-4241-8929-2e058bfbffbe\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dbs8c" Jul 06 19:34:00.379200 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:00.379160 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dbs8c" Jul 06 19:34:00.461541 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:00.461513 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kjf75"] Jul 06 19:34:00.466698 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:00.466628 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kjf75" Jul 06 19:34:00.480314 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:00.480292 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kjf75"] Jul 06 19:34:00.514985 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:00.514951 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9696\" (UniqueName: \"kubernetes.io/projected/3b2b2fce-db68-4292-bc62-08e5d48d2c3f-kube-api-access-s9696\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kjf75\" (UID: \"3b2b2fce-db68-4292-bc62-08e5d48d2c3f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kjf75" Jul 06 19:34:00.515232 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:00.515014 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b2b2fce-db68-4292-bc62-08e5d48d2c3f-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kjf75\" (UID: \"3b2b2fce-db68-4292-bc62-08e5d48d2c3f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kjf75" Jul 06 19:34:00.515232 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:00.515133 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b2b2fce-db68-4292-bc62-08e5d48d2c3f-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kjf75\" (UID: \"3b2b2fce-db68-4292-bc62-08e5d48d2c3f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kjf75" Jul 06 19:34:00.517764 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:00.517745 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dbs8c"] Jul 06 19:34:00.519813 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:34:00.519790 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb57645a_f2c3_4241_8929_2e058bfbffbe.slice/crio-bf9cf9dc68f745bc528eec46674194d7b955430bd81ae00207dee6baa04f8ff4 WatchSource:0}: Error finding container bf9cf9dc68f745bc528eec46674194d7b955430bd81ae00207dee6baa04f8ff4: Status 404 returned error can't find the container with id bf9cf9dc68f745bc528eec46674194d7b955430bd81ae00207dee6baa04f8ff4 Jul 06 19:34:00.616257 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:00.616234 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9696\" (UniqueName: \"kubernetes.io/projected/3b2b2fce-db68-4292-bc62-08e5d48d2c3f-kube-api-access-s9696\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kjf75\" (UID: \"3b2b2fce-db68-4292-bc62-08e5d48d2c3f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kjf75" Jul 06 19:34:00.616367 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:00.616271 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b2b2fce-db68-4292-bc62-08e5d48d2c3f-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kjf75\" (UID: \"3b2b2fce-db68-4292-bc62-08e5d48d2c3f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kjf75" Jul 06 19:34:00.616367 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:00.616322 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b2b2fce-db68-4292-bc62-08e5d48d2c3f-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kjf75\" (UID: \"3b2b2fce-db68-4292-bc62-08e5d48d2c3f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kjf75" Jul 06 19:34:00.616619 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:00.616604 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b2b2fce-db68-4292-bc62-08e5d48d2c3f-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kjf75\" (UID: \"3b2b2fce-db68-4292-bc62-08e5d48d2c3f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kjf75" Jul 06 19:34:00.616692 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:00.616667 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b2b2fce-db68-4292-bc62-08e5d48d2c3f-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kjf75\" (UID: \"3b2b2fce-db68-4292-bc62-08e5d48d2c3f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kjf75" Jul 06 19:34:00.626461 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:00.626441 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9696\" (UniqueName: \"kubernetes.io/projected/3b2b2fce-db68-4292-bc62-08e5d48d2c3f-kube-api-access-s9696\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kjf75\" (UID: \"3b2b2fce-db68-4292-bc62-08e5d48d2c3f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kjf75" Jul 06 19:34:00.776835 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:00.776785 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kjf75" Jul 06 19:34:00.867112 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:00.867085 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08d46k"] Jul 06 19:34:00.872025 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:00.872005 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08d46k" Jul 06 19:34:00.881532 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:00.881513 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08d46k"] Jul 06 19:34:00.901643 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:00.901621 2575 generic.go:356] "Generic (PLEG): container finished" podID="cb57645a-f2c3-4241-8929-2e058bfbffbe" containerID="f458e62c90df0f8ba437039ed9f90306d493cdc9c00b4c4a88101e8206166a08" exitCode=0 Jul 06 19:34:00.901752 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:00.901698 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dbs8c" event={"ID":"cb57645a-f2c3-4241-8929-2e058bfbffbe","Type":"ContainerDied","Data":"f458e62c90df0f8ba437039ed9f90306d493cdc9c00b4c4a88101e8206166a08"} Jul 06 19:34:00.901752 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:00.901743 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dbs8c" event={"ID":"cb57645a-f2c3-4241-8929-2e058bfbffbe","Type":"ContainerStarted","Data":"bf9cf9dc68f745bc528eec46674194d7b955430bd81ae00207dee6baa04f8ff4"} Jul 06 19:34:00.912456 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:00.912436 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kjf75"] Jul 06 19:34:00.915496 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:34:00.915472 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b2b2fce_db68_4292_bc62_08e5d48d2c3f.slice/crio-0ac83dea1fe559ab4fd4a201536a0d269541870de84ba3264db6d4f50b9b89d4 WatchSource:0}: Error finding container 0ac83dea1fe559ab4fd4a201536a0d269541870de84ba3264db6d4f50b9b89d4: Status 404 returned error can't find the container with id 0ac83dea1fe559ab4fd4a201536a0d269541870de84ba3264db6d4f50b9b89d4 Jul 06 19:34:00.918175 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:00.918158 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2fv5\" (UniqueName: \"kubernetes.io/projected/f77e4ac6-70cf-4dc0-8829-740324b3f385-kube-api-access-t2fv5\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08d46k\" (UID: \"f77e4ac6-70cf-4dc0-8829-740324b3f385\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08d46k" Jul 06 19:34:00.918268 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:00.918250 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f77e4ac6-70cf-4dc0-8829-740324b3f385-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08d46k\" (UID: \"f77e4ac6-70cf-4dc0-8829-740324b3f385\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08d46k" Jul 06 19:34:00.918322 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:00.918311 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f77e4ac6-70cf-4dc0-8829-740324b3f385-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08d46k\" (UID: \"f77e4ac6-70cf-4dc0-8829-740324b3f385\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08d46k" Jul 06 19:34:01.019648 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:01.019622 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f77e4ac6-70cf-4dc0-8829-740324b3f385-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08d46k\" (UID: \"f77e4ac6-70cf-4dc0-8829-740324b3f385\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08d46k" Jul 06 19:34:01.019871 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:01.019708 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2fv5\" (UniqueName: \"kubernetes.io/projected/f77e4ac6-70cf-4dc0-8829-740324b3f385-kube-api-access-t2fv5\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08d46k\" (UID: \"f77e4ac6-70cf-4dc0-8829-740324b3f385\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08d46k" Jul 06 19:34:01.019871 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:01.019771 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f77e4ac6-70cf-4dc0-8829-740324b3f385-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08d46k\" (UID: \"f77e4ac6-70cf-4dc0-8829-740324b3f385\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08d46k" Jul 06 19:34:01.020026 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:01.020004 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f77e4ac6-70cf-4dc0-8829-740324b3f385-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08d46k\" (UID: \"f77e4ac6-70cf-4dc0-8829-740324b3f385\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08d46k" Jul 06 19:34:01.020111 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:01.020088 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f77e4ac6-70cf-4dc0-8829-740324b3f385-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08d46k\" (UID: \"f77e4ac6-70cf-4dc0-8829-740324b3f385\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08d46k" Jul 06 19:34:01.034137 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:01.034094 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2fv5\" (UniqueName: \"kubernetes.io/projected/f77e4ac6-70cf-4dc0-8829-740324b3f385-kube-api-access-t2fv5\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08d46k\" (UID: \"f77e4ac6-70cf-4dc0-8829-740324b3f385\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08d46k" Jul 06 19:34:01.183224 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:01.183201 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08d46k" Jul 06 19:34:01.268659 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:01.268630 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gq9fm"] Jul 06 19:34:01.273751 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:01.273729 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gq9fm" Jul 06 19:34:01.281551 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:01.281518 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gq9fm"] Jul 06 19:34:01.323291 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:01.323267 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a90a1c3-1bff-4393-a787-f78b5e0e367f-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gq9fm\" (UID: \"6a90a1c3-1bff-4393-a787-f78b5e0e367f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gq9fm" Jul 06 19:34:01.323417 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:01.323302 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a90a1c3-1bff-4393-a787-f78b5e0e367f-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gq9fm\" (UID: \"6a90a1c3-1bff-4393-a787-f78b5e0e367f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gq9fm" Jul 06 19:34:01.323417 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:01.323333 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqz77\" (UniqueName: \"kubernetes.io/projected/6a90a1c3-1bff-4393-a787-f78b5e0e367f-kube-api-access-qqz77\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gq9fm\" (UID: \"6a90a1c3-1bff-4393-a787-f78b5e0e367f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gq9fm" Jul 06 19:34:01.325686 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:01.325668 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08d46k"] Jul 06 19:34:01.328285 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:34:01.328255 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf77e4ac6_70cf_4dc0_8829_740324b3f385.slice/crio-85f18fa1154b77054a1ac2c5d556f9c1faebe6e78dd61c647538a9c28cd11d52 WatchSource:0}: Error finding container 85f18fa1154b77054a1ac2c5d556f9c1faebe6e78dd61c647538a9c28cd11d52: Status 404 returned error can't find the container with id 85f18fa1154b77054a1ac2c5d556f9c1faebe6e78dd61c647538a9c28cd11d52 Jul 06 19:34:01.424387 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:01.424359 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqz77\" (UniqueName: \"kubernetes.io/projected/6a90a1c3-1bff-4393-a787-f78b5e0e367f-kube-api-access-qqz77\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gq9fm\" (UID: \"6a90a1c3-1bff-4393-a787-f78b5e0e367f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gq9fm" Jul 06 19:34:01.424522 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:01.424510 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a90a1c3-1bff-4393-a787-f78b5e0e367f-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gq9fm\" (UID: \"6a90a1c3-1bff-4393-a787-f78b5e0e367f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gq9fm" Jul 06 19:34:01.424594 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:01.424550 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a90a1c3-1bff-4393-a787-f78b5e0e367f-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gq9fm\" (UID: \"6a90a1c3-1bff-4393-a787-f78b5e0e367f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gq9fm" Jul 06 19:34:01.424859 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:01.424841 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a90a1c3-1bff-4393-a787-f78b5e0e367f-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gq9fm\" (UID: \"6a90a1c3-1bff-4393-a787-f78b5e0e367f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gq9fm" Jul 06 19:34:01.424945 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:01.424871 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a90a1c3-1bff-4393-a787-f78b5e0e367f-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gq9fm\" (UID: \"6a90a1c3-1bff-4393-a787-f78b5e0e367f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gq9fm" Jul 06 19:34:01.433946 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:01.433924 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqz77\" (UniqueName: \"kubernetes.io/projected/6a90a1c3-1bff-4393-a787-f78b5e0e367f-kube-api-access-qqz77\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gq9fm\" (UID: \"6a90a1c3-1bff-4393-a787-f78b5e0e367f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gq9fm" Jul 06 19:34:01.584750 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:01.584690 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gq9fm" Jul 06 19:34:01.721689 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:01.721670 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gq9fm"] Jul 06 19:34:01.723552 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:34:01.723530 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a90a1c3_1bff_4393_a787_f78b5e0e367f.slice/crio-7a6ef05a48074983803b511083bb2fd1e6bd679b4baf2302352476b98dc73476 WatchSource:0}: Error finding container 7a6ef05a48074983803b511083bb2fd1e6bd679b4baf2302352476b98dc73476: Status 404 returned error can't find the container with id 7a6ef05a48074983803b511083bb2fd1e6bd679b4baf2302352476b98dc73476 Jul 06 19:34:01.908763 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:01.908731 2575 generic.go:356] "Generic (PLEG): container finished" podID="3b2b2fce-db68-4292-bc62-08e5d48d2c3f" containerID="7fc3708b8e68d98da20850e0a7224ddc990810d3bce6765a9db159b43e9d2df2" exitCode=0 Jul 06 19:34:01.908928 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:01.908811 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kjf75" event={"ID":"3b2b2fce-db68-4292-bc62-08e5d48d2c3f","Type":"ContainerDied","Data":"7fc3708b8e68d98da20850e0a7224ddc990810d3bce6765a9db159b43e9d2df2"} Jul 06 19:34:01.908928 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:01.908862 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kjf75" event={"ID":"3b2b2fce-db68-4292-bc62-08e5d48d2c3f","Type":"ContainerStarted","Data":"0ac83dea1fe559ab4fd4a201536a0d269541870de84ba3264db6d4f50b9b89d4"} Jul 06 19:34:01.910348 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:01.910325 2575 generic.go:356] "Generic (PLEG): container finished" podID="6a90a1c3-1bff-4393-a787-f78b5e0e367f" containerID="5b04cdd44eb231c031639252ff7c609bb706ca53b0c8a4dbd5115523e7fe8510" exitCode=0 Jul 06 19:34:01.910436 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:01.910385 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gq9fm" event={"ID":"6a90a1c3-1bff-4393-a787-f78b5e0e367f","Type":"ContainerDied","Data":"5b04cdd44eb231c031639252ff7c609bb706ca53b0c8a4dbd5115523e7fe8510"} Jul 06 19:34:01.910436 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:01.910418 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gq9fm" event={"ID":"6a90a1c3-1bff-4393-a787-f78b5e0e367f","Type":"ContainerStarted","Data":"7a6ef05a48074983803b511083bb2fd1e6bd679b4baf2302352476b98dc73476"} Jul 06 19:34:01.911912 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:01.911889 2575 generic.go:356] "Generic (PLEG): container finished" podID="f77e4ac6-70cf-4dc0-8829-740324b3f385" containerID="81fc992f04e4792952d84b4c107066e3e06ba9aee412e04730e31aa01d65531a" exitCode=0 Jul 06 19:34:01.912000 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:01.911980 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08d46k" event={"ID":"f77e4ac6-70cf-4dc0-8829-740324b3f385","Type":"ContainerDied","Data":"81fc992f04e4792952d84b4c107066e3e06ba9aee412e04730e31aa01d65531a"} Jul 06 19:34:01.912057 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:01.912021 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08d46k" event={"ID":"f77e4ac6-70cf-4dc0-8829-740324b3f385","Type":"ContainerStarted","Data":"85f18fa1154b77054a1ac2c5d556f9c1faebe6e78dd61c647538a9c28cd11d52"} Jul 06 19:34:01.914015 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:01.913949 2575 generic.go:356] "Generic (PLEG): container finished" podID="cb57645a-f2c3-4241-8929-2e058bfbffbe" containerID="ad5521b31688b18961d55c7994f467aa116bd40aa696533dfc5aeb8ef05c3752" exitCode=0 Jul 06 19:34:01.914015 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:01.914002 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dbs8c" event={"ID":"cb57645a-f2c3-4241-8929-2e058bfbffbe","Type":"ContainerDied","Data":"ad5521b31688b18961d55c7994f467aa116bd40aa696533dfc5aeb8ef05c3752"} Jul 06 19:34:02.923582 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:02.923520 2575 generic.go:356] "Generic (PLEG): container finished" podID="3b2b2fce-db68-4292-bc62-08e5d48d2c3f" containerID="c185215fc6c81a9e6a92e678ba21529c7b421d9805d8164ae401280085069116" exitCode=0 Jul 06 19:34:02.923893 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:02.923577 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kjf75" event={"ID":"3b2b2fce-db68-4292-bc62-08e5d48d2c3f","Type":"ContainerDied","Data":"c185215fc6c81a9e6a92e678ba21529c7b421d9805d8164ae401280085069116"} Jul 06 19:34:02.925456 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:02.925435 2575 generic.go:356] "Generic (PLEG): container finished" podID="6a90a1c3-1bff-4393-a787-f78b5e0e367f" containerID="a5a0dba5fcd9f05d6cc06d4e9d0e7756c341e4515aa9ca7908a7a86c620bb986" exitCode=0 Jul 06 19:34:02.925542 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:02.925515 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gq9fm" event={"ID":"6a90a1c3-1bff-4393-a787-f78b5e0e367f","Type":"ContainerDied","Data":"a5a0dba5fcd9f05d6cc06d4e9d0e7756c341e4515aa9ca7908a7a86c620bb986"} Jul 06 19:34:02.927372 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:02.927353 2575 generic.go:356] "Generic (PLEG): container finished" podID="f77e4ac6-70cf-4dc0-8829-740324b3f385" containerID="8c2dcf1fbca20bf808daeac155747250e15c8a2b8dde219fe95032d82a9cbc0e" exitCode=0 Jul 06 19:34:02.927469 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:02.927428 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08d46k" event={"ID":"f77e4ac6-70cf-4dc0-8829-740324b3f385","Type":"ContainerDied","Data":"8c2dcf1fbca20bf808daeac155747250e15c8a2b8dde219fe95032d82a9cbc0e"} Jul 06 19:34:02.929745 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:02.929728 2575 generic.go:356] "Generic (PLEG): container finished" podID="cb57645a-f2c3-4241-8929-2e058bfbffbe" containerID="331c759de6f2071e798f087e646a22edd531ee30983ded1c61c2c899896d9213" exitCode=0 Jul 06 19:34:02.929842 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:02.929828 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dbs8c" event={"ID":"cb57645a-f2c3-4241-8929-2e058bfbffbe","Type":"ContainerDied","Data":"331c759de6f2071e798f087e646a22edd531ee30983ded1c61c2c899896d9213"} Jul 06 19:34:03.943897 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:03.943867 2575 generic.go:356] "Generic (PLEG): container finished" podID="3b2b2fce-db68-4292-bc62-08e5d48d2c3f" containerID="2e8e4220e1a19513c4bdb03e125a43e2017f263b16d7dde48b05b4cb69d82ec9" exitCode=0 Jul 06 19:34:03.944339 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:03.943956 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kjf75" event={"ID":"3b2b2fce-db68-4292-bc62-08e5d48d2c3f","Type":"ContainerDied","Data":"2e8e4220e1a19513c4bdb03e125a43e2017f263b16d7dde48b05b4cb69d82ec9"} Jul 06 19:34:03.945837 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:03.945814 2575 generic.go:356] "Generic (PLEG): container finished" podID="6a90a1c3-1bff-4393-a787-f78b5e0e367f" containerID="0c92d82d4cd20297235ebb622e8fa508d9f01cd79dd286593bcc3c474660a98a" exitCode=0 Jul 06 19:34:03.945950 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:03.945878 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gq9fm" event={"ID":"6a90a1c3-1bff-4393-a787-f78b5e0e367f","Type":"ContainerDied","Data":"0c92d82d4cd20297235ebb622e8fa508d9f01cd79dd286593bcc3c474660a98a"} Jul 06 19:34:03.947697 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:03.947669 2575 generic.go:356] "Generic (PLEG): container finished" podID="f77e4ac6-70cf-4dc0-8829-740324b3f385" containerID="3bea1668ac9b1bb44ccdd1e3873ea49a40b085a24f44da6f1b7487490a8dd916" exitCode=0 Jul 06 19:34:03.947804 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:03.947687 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08d46k" event={"ID":"f77e4ac6-70cf-4dc0-8829-740324b3f385","Type":"ContainerDied","Data":"3bea1668ac9b1bb44ccdd1e3873ea49a40b085a24f44da6f1b7487490a8dd916"} Jul 06 19:34:04.079168 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:04.079139 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dbs8c" Jul 06 19:34:04.146582 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:04.146558 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb57645a-f2c3-4241-8929-2e058bfbffbe-bundle\") pod \"cb57645a-f2c3-4241-8929-2e058bfbffbe\" (UID: \"cb57645a-f2c3-4241-8929-2e058bfbffbe\") " Jul 06 19:34:04.146710 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:04.146625 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb57645a-f2c3-4241-8929-2e058bfbffbe-util\") pod \"cb57645a-f2c3-4241-8929-2e058bfbffbe\" (UID: \"cb57645a-f2c3-4241-8929-2e058bfbffbe\") " Jul 06 19:34:04.146710 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:04.146661 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5fvq\" (UniqueName: \"kubernetes.io/projected/cb57645a-f2c3-4241-8929-2e058bfbffbe-kube-api-access-x5fvq\") pod \"cb57645a-f2c3-4241-8929-2e058bfbffbe\" (UID: \"cb57645a-f2c3-4241-8929-2e058bfbffbe\") " Jul 06 19:34:04.147077 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:04.147050 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb57645a-f2c3-4241-8929-2e058bfbffbe-bundle" (OuterVolumeSpecName: "bundle") pod "cb57645a-f2c3-4241-8929-2e058bfbffbe" (UID: "cb57645a-f2c3-4241-8929-2e058bfbffbe"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:34:04.148640 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:04.148616 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb57645a-f2c3-4241-8929-2e058bfbffbe-kube-api-access-x5fvq" (OuterVolumeSpecName: "kube-api-access-x5fvq") pod "cb57645a-f2c3-4241-8929-2e058bfbffbe" (UID: "cb57645a-f2c3-4241-8929-2e058bfbffbe"). InnerVolumeSpecName "kube-api-access-x5fvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:34:04.151842 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:04.151823 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb57645a-f2c3-4241-8929-2e058bfbffbe-util" (OuterVolumeSpecName: "util") pod "cb57645a-f2c3-4241-8929-2e058bfbffbe" (UID: "cb57645a-f2c3-4241-8929-2e058bfbffbe"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:34:04.247740 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:04.247654 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb57645a-f2c3-4241-8929-2e058bfbffbe-bundle\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:34:04.247740 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:04.247692 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb57645a-f2c3-4241-8929-2e058bfbffbe-util\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:34:04.247740 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:04.247702 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5fvq\" (UniqueName: \"kubernetes.io/projected/cb57645a-f2c3-4241-8929-2e058bfbffbe-kube-api-access-x5fvq\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:34:04.956151 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:04.956120 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dbs8c" Jul 06 19:34:04.956546 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:04.956168 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dbs8c" event={"ID":"cb57645a-f2c3-4241-8929-2e058bfbffbe","Type":"ContainerDied","Data":"bf9cf9dc68f745bc528eec46674194d7b955430bd81ae00207dee6baa04f8ff4"} Jul 06 19:34:04.956546 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:04.956229 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf9cf9dc68f745bc528eec46674194d7b955430bd81ae00207dee6baa04f8ff4" Jul 06 19:34:05.103141 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:05.103116 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gq9fm" Jul 06 19:34:05.139759 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:05.139736 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kjf75" Jul 06 19:34:05.142775 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:05.142758 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08d46k" Jul 06 19:34:05.154325 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:05.154307 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqz77\" (UniqueName: \"kubernetes.io/projected/6a90a1c3-1bff-4393-a787-f78b5e0e367f-kube-api-access-qqz77\") pod \"6a90a1c3-1bff-4393-a787-f78b5e0e367f\" (UID: \"6a90a1c3-1bff-4393-a787-f78b5e0e367f\") " Jul 06 19:34:05.154414 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:05.154336 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a90a1c3-1bff-4393-a787-f78b5e0e367f-util\") pod \"6a90a1c3-1bff-4393-a787-f78b5e0e367f\" (UID: \"6a90a1c3-1bff-4393-a787-f78b5e0e367f\") " Jul 06 19:34:05.154414 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:05.154403 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a90a1c3-1bff-4393-a787-f78b5e0e367f-bundle\") pod \"6a90a1c3-1bff-4393-a787-f78b5e0e367f\" (UID: \"6a90a1c3-1bff-4393-a787-f78b5e0e367f\") " Jul 06 19:34:05.155115 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:05.155088 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a90a1c3-1bff-4393-a787-f78b5e0e367f-bundle" (OuterVolumeSpecName: "bundle") pod "6a90a1c3-1bff-4393-a787-f78b5e0e367f" (UID: "6a90a1c3-1bff-4393-a787-f78b5e0e367f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:34:05.156263 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:05.156244 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a90a1c3-1bff-4393-a787-f78b5e0e367f-kube-api-access-qqz77" (OuterVolumeSpecName: "kube-api-access-qqz77") pod "6a90a1c3-1bff-4393-a787-f78b5e0e367f" (UID: "6a90a1c3-1bff-4393-a787-f78b5e0e367f"). InnerVolumeSpecName "kube-api-access-qqz77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:34:05.159452 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:05.159434 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a90a1c3-1bff-4393-a787-f78b5e0e367f-util" (OuterVolumeSpecName: "util") pod "6a90a1c3-1bff-4393-a787-f78b5e0e367f" (UID: "6a90a1c3-1bff-4393-a787-f78b5e0e367f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:34:05.255251 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:05.255176 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9696\" (UniqueName: \"kubernetes.io/projected/3b2b2fce-db68-4292-bc62-08e5d48d2c3f-kube-api-access-s9696\") pod \"3b2b2fce-db68-4292-bc62-08e5d48d2c3f\" (UID: \"3b2b2fce-db68-4292-bc62-08e5d48d2c3f\") " Jul 06 19:34:05.255251 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:05.255236 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f77e4ac6-70cf-4dc0-8829-740324b3f385-util\") pod \"f77e4ac6-70cf-4dc0-8829-740324b3f385\" (UID: \"f77e4ac6-70cf-4dc0-8829-740324b3f385\") " Jul 06 19:34:05.255428 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:05.255262 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b2b2fce-db68-4292-bc62-08e5d48d2c3f-bundle\") pod \"3b2b2fce-db68-4292-bc62-08e5d48d2c3f\" (UID: \"3b2b2fce-db68-4292-bc62-08e5d48d2c3f\") " Jul 06 19:34:05.255428 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:05.255292 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b2b2fce-db68-4292-bc62-08e5d48d2c3f-util\") pod \"3b2b2fce-db68-4292-bc62-08e5d48d2c3f\" (UID: \"3b2b2fce-db68-4292-bc62-08e5d48d2c3f\") " Jul 06 19:34:05.255428 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:05.255337 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f77e4ac6-70cf-4dc0-8829-740324b3f385-bundle\") pod \"f77e4ac6-70cf-4dc0-8829-740324b3f385\" (UID: \"f77e4ac6-70cf-4dc0-8829-740324b3f385\") " Jul 06 19:34:05.255428 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:05.255377 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2fv5\" (UniqueName: \"kubernetes.io/projected/f77e4ac6-70cf-4dc0-8829-740324b3f385-kube-api-access-t2fv5\") pod \"f77e4ac6-70cf-4dc0-8829-740324b3f385\" (UID: \"f77e4ac6-70cf-4dc0-8829-740324b3f385\") " Jul 06 19:34:05.255712 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:05.255692 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqz77\" (UniqueName: \"kubernetes.io/projected/6a90a1c3-1bff-4393-a787-f78b5e0e367f-kube-api-access-qqz77\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:34:05.255775 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:05.255716 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a90a1c3-1bff-4393-a787-f78b5e0e367f-util\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:34:05.255775 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:05.255730 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a90a1c3-1bff-4393-a787-f78b5e0e367f-bundle\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:34:05.255888 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:05.255860 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f77e4ac6-70cf-4dc0-8829-740324b3f385-bundle" (OuterVolumeSpecName: "bundle") pod "f77e4ac6-70cf-4dc0-8829-740324b3f385" (UID: "f77e4ac6-70cf-4dc0-8829-740324b3f385"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:34:05.255930 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:05.255867 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b2b2fce-db68-4292-bc62-08e5d48d2c3f-bundle" (OuterVolumeSpecName: "bundle") pod "3b2b2fce-db68-4292-bc62-08e5d48d2c3f" (UID: "3b2b2fce-db68-4292-bc62-08e5d48d2c3f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:34:05.257745 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:05.257722 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f77e4ac6-70cf-4dc0-8829-740324b3f385-kube-api-access-t2fv5" (OuterVolumeSpecName: "kube-api-access-t2fv5") pod "f77e4ac6-70cf-4dc0-8829-740324b3f385" (UID: "f77e4ac6-70cf-4dc0-8829-740324b3f385"). InnerVolumeSpecName "kube-api-access-t2fv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:34:05.257840 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:05.257754 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b2b2fce-db68-4292-bc62-08e5d48d2c3f-kube-api-access-s9696" (OuterVolumeSpecName: "kube-api-access-s9696") pod "3b2b2fce-db68-4292-bc62-08e5d48d2c3f" (UID: "3b2b2fce-db68-4292-bc62-08e5d48d2c3f"). InnerVolumeSpecName "kube-api-access-s9696". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:34:05.261168 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:05.261146 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f77e4ac6-70cf-4dc0-8829-740324b3f385-util" (OuterVolumeSpecName: "util") pod "f77e4ac6-70cf-4dc0-8829-740324b3f385" (UID: "f77e4ac6-70cf-4dc0-8829-740324b3f385"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:34:05.261785 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:05.261767 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b2b2fce-db68-4292-bc62-08e5d48d2c3f-util" (OuterVolumeSpecName: "util") pod "3b2b2fce-db68-4292-bc62-08e5d48d2c3f" (UID: "3b2b2fce-db68-4292-bc62-08e5d48d2c3f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:34:05.356821 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:05.356797 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2fv5\" (UniqueName: \"kubernetes.io/projected/f77e4ac6-70cf-4dc0-8829-740324b3f385-kube-api-access-t2fv5\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:34:05.356821 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:05.356826 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9696\" (UniqueName: \"kubernetes.io/projected/3b2b2fce-db68-4292-bc62-08e5d48d2c3f-kube-api-access-s9696\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:34:05.356963 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:05.356840 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f77e4ac6-70cf-4dc0-8829-740324b3f385-util\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:34:05.356963 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:05.356853 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b2b2fce-db68-4292-bc62-08e5d48d2c3f-bundle\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:34:05.356963 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:05.356864 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b2b2fce-db68-4292-bc62-08e5d48d2c3f-util\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:34:05.356963 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:05.356875 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f77e4ac6-70cf-4dc0-8829-740324b3f385-bundle\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:34:05.963362 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:05.963329 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gq9fm" event={"ID":"6a90a1c3-1bff-4393-a787-f78b5e0e367f","Type":"ContainerDied","Data":"7a6ef05a48074983803b511083bb2fd1e6bd679b4baf2302352476b98dc73476"} Jul 06 19:34:05.963774 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:05.963381 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a6ef05a48074983803b511083bb2fd1e6bd679b4baf2302352476b98dc73476" Jul 06 19:34:05.963774 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:05.963344 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gq9fm" Jul 06 19:34:05.965438 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:05.965409 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08d46k" event={"ID":"f77e4ac6-70cf-4dc0-8829-740324b3f385","Type":"ContainerDied","Data":"85f18fa1154b77054a1ac2c5d556f9c1faebe6e78dd61c647538a9c28cd11d52"} Jul 06 19:34:05.965564 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:05.965455 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85f18fa1154b77054a1ac2c5d556f9c1faebe6e78dd61c647538a9c28cd11d52" Jul 06 19:34:05.965564 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:05.965418 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08d46k" Jul 06 19:34:05.967556 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:05.967536 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kjf75" event={"ID":"3b2b2fce-db68-4292-bc62-08e5d48d2c3f","Type":"ContainerDied","Data":"0ac83dea1fe559ab4fd4a201536a0d269541870de84ba3264db6d4f50b9b89d4"} Jul 06 19:34:05.967661 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:05.967566 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ac83dea1fe559ab4fd4a201536a0d269541870de84ba3264db6d4f50b9b89d4" Jul 06 19:34:05.967661 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:05.967586 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kjf75" Jul 06 19:34:10.446949 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:10.446909 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-8xjts"] Jul 06 19:34:10.447371 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:10.447355 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb57645a-f2c3-4241-8929-2e058bfbffbe" containerName="extract" Jul 06 19:34:10.447410 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:10.447373 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb57645a-f2c3-4241-8929-2e058bfbffbe" containerName="extract" Jul 06 19:34:10.447410 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:10.447385 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b2b2fce-db68-4292-bc62-08e5d48d2c3f" containerName="util" Jul 06 19:34:10.447410 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:10.447392 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b2b2fce-db68-4292-bc62-08e5d48d2c3f" containerName="util" Jul 06 19:34:10.447410 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:10.447398 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a90a1c3-1bff-4393-a787-f78b5e0e367f" containerName="util" Jul 06 19:34:10.447410 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:10.447404 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a90a1c3-1bff-4393-a787-f78b5e0e367f" containerName="util" Jul 06 19:34:10.447410 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:10.447411 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a90a1c3-1bff-4393-a787-f78b5e0e367f" containerName="extract" Jul 06 19:34:10.447599 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:10.447417 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a90a1c3-1bff-4393-a787-f78b5e0e367f" containerName="extract" Jul 06 19:34:10.447599 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:10.447424 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb57645a-f2c3-4241-8929-2e058bfbffbe" containerName="pull" Jul 06 19:34:10.447599 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:10.447432 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb57645a-f2c3-4241-8929-2e058bfbffbe" containerName="pull" Jul 06 19:34:10.447599 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:10.447446 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a90a1c3-1bff-4393-a787-f78b5e0e367f" containerName="pull" Jul 06 19:34:10.447599 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:10.447451 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a90a1c3-1bff-4393-a787-f78b5e0e367f" containerName="pull" Jul 06 19:34:10.447599 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:10.447457 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb57645a-f2c3-4241-8929-2e058bfbffbe" containerName="util" Jul 06 19:34:10.447599 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:10.447463 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb57645a-f2c3-4241-8929-2e058bfbffbe" containerName="util" Jul 06 19:34:10.447599 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:10.447468 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b2b2fce-db68-4292-bc62-08e5d48d2c3f" containerName="pull" Jul 06 19:34:10.447599 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:10.447473 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b2b2fce-db68-4292-bc62-08e5d48d2c3f" containerName="pull" Jul 06 19:34:10.447599 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:10.447487 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f77e4ac6-70cf-4dc0-8829-740324b3f385" containerName="util" Jul 06 19:34:10.447599 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:10.447491 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f77e4ac6-70cf-4dc0-8829-740324b3f385" containerName="util" Jul 06 19:34:10.447599 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:10.447496 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b2b2fce-db68-4292-bc62-08e5d48d2c3f" containerName="extract" Jul 06 19:34:10.447599 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:10.447501 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b2b2fce-db68-4292-bc62-08e5d48d2c3f" containerName="extract" Jul 06 19:34:10.447599 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:10.447511 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f77e4ac6-70cf-4dc0-8829-740324b3f385" containerName="extract" Jul 06 19:34:10.447599 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:10.447518 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f77e4ac6-70cf-4dc0-8829-740324b3f385" containerName="extract" Jul 06 19:34:10.447599 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:10.447526 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f77e4ac6-70cf-4dc0-8829-740324b3f385" containerName="pull" Jul 06 19:34:10.447599 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:10.447530 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f77e4ac6-70cf-4dc0-8829-740324b3f385" containerName="pull" Jul 06 19:34:10.447599 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:10.447591 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b2b2fce-db68-4292-bc62-08e5d48d2c3f" containerName="extract" Jul 06 19:34:10.447599 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:10.447602 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a90a1c3-1bff-4393-a787-f78b5e0e367f" containerName="extract" Jul 06 19:34:10.447599 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:10.447607 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb57645a-f2c3-4241-8929-2e058bfbffbe" containerName="extract" Jul 06 19:34:10.448115 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:10.447614 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f77e4ac6-70cf-4dc0-8829-740324b3f385" containerName="extract" Jul 06 19:34:10.452024 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:10.452006 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8xjts" Jul 06 19:34:10.455228 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:10.455207 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 06 19:34:10.455331 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:10.455231 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-2wvcv\"" Jul 06 19:34:10.471108 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:10.471085 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-8xjts"] Jul 06 19:34:10.496427 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:10.496403 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wszlx\" (UniqueName: \"kubernetes.io/projected/e692ea80-46aa-4916-a536-ce492c375f61-kube-api-access-wszlx\") pod \"dns-operator-controller-manager-864dd78c4c-8xjts\" (UID: \"e692ea80-46aa-4916-a536-ce492c375f61\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8xjts" Jul 06 19:34:10.597324 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:10.597302 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wszlx\" (UniqueName: \"kubernetes.io/projected/e692ea80-46aa-4916-a536-ce492c375f61-kube-api-access-wszlx\") pod \"dns-operator-controller-manager-864dd78c4c-8xjts\" (UID: \"e692ea80-46aa-4916-a536-ce492c375f61\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8xjts" Jul 06 19:34:10.615128 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:10.615092 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wszlx\" (UniqueName: \"kubernetes.io/projected/e692ea80-46aa-4916-a536-ce492c375f61-kube-api-access-wszlx\") pod \"dns-operator-controller-manager-864dd78c4c-8xjts\" (UID: \"e692ea80-46aa-4916-a536-ce492c375f61\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8xjts" Jul 06 19:34:10.762313 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:10.762262 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8xjts" Jul 06 19:34:10.903667 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:10.903646 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-8xjts"] Jul 06 19:34:10.905532 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:34:10.905506 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode692ea80_46aa_4916_a536_ce492c375f61.slice/crio-0f1c69cb7e8b04b8c0fde05b4a5e1584ec37634d1e6b0b6bd8d253c0945b3c3c WatchSource:0}: Error finding container 0f1c69cb7e8b04b8c0fde05b4a5e1584ec37634d1e6b0b6bd8d253c0945b3c3c: Status 404 returned error can't find the container with id 0f1c69cb7e8b04b8c0fde05b4a5e1584ec37634d1e6b0b6bd8d253c0945b3c3c Jul 06 19:34:10.995334 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:10.995304 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8xjts" event={"ID":"e692ea80-46aa-4916-a536-ce492c375f61","Type":"ContainerStarted","Data":"0f1c69cb7e8b04b8c0fde05b4a5e1584ec37634d1e6b0b6bd8d253c0945b3c3c"} Jul 06 19:34:13.010393 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:13.010351 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8xjts" event={"ID":"e692ea80-46aa-4916-a536-ce492c375f61","Type":"ContainerStarted","Data":"0dee66a9400007351fe5737d7484137056763fb8e2ac7b6dcb4bf6bee1259338"} Jul 06 19:34:13.010852 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:13.010466 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8xjts" Jul 06 19:34:13.035059 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:13.034997 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8xjts" podStartSLOduration=1.375878046 podStartE2EDuration="3.034984151s" totalImagesPullingTime="1.659106105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:34:13.033662408 +0000 UTC m=+461.853751121" watchObservedRunningTime="2026-07-06 19:34:13.034984151 +0000 UTC m=+461.855072862" Jul 06 19:34:13.993076 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:13.993045 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sfvcl"] Jul 06 19:34:13.996666 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:13.996644 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sfvcl" Jul 06 19:34:14.001382 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:14.001366 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-h8qtn\"" Jul 06 19:34:14.013951 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:14.013928 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sfvcl"] Jul 06 19:34:14.135728 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:14.135699 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r75zm\" (UniqueName: \"kubernetes.io/projected/5019d9af-5f01-470e-8a72-0a062617aa21-kube-api-access-r75zm\") pod \"limitador-operator-controller-manager-859d7b78bd-sfvcl\" (UID: \"5019d9af-5f01-470e-8a72-0a062617aa21\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sfvcl" Jul 06 19:34:14.237265 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:14.237236 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r75zm\" (UniqueName: \"kubernetes.io/projected/5019d9af-5f01-470e-8a72-0a062617aa21-kube-api-access-r75zm\") pod \"limitador-operator-controller-manager-859d7b78bd-sfvcl\" (UID: \"5019d9af-5f01-470e-8a72-0a062617aa21\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sfvcl" Jul 06 19:34:14.255688 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:14.255624 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r75zm\" (UniqueName: \"kubernetes.io/projected/5019d9af-5f01-470e-8a72-0a062617aa21-kube-api-access-r75zm\") pod \"limitador-operator-controller-manager-859d7b78bd-sfvcl\" (UID: \"5019d9af-5f01-470e-8a72-0a062617aa21\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sfvcl" Jul 06 19:34:14.306574 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:14.306552 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sfvcl" Jul 06 19:34:14.440709 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:14.440688 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sfvcl"] Jul 06 19:34:14.442345 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:34:14.442321 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5019d9af_5f01_470e_8a72_0a062617aa21.slice/crio-924b9871650eae8ae8b67179c28f65c60bdbaebe0c28d21f7a996c66ae439cff WatchSource:0}: Error finding container 924b9871650eae8ae8b67179c28f65c60bdbaebe0c28d21f7a996c66ae439cff: Status 404 returned error can't find the container with id 924b9871650eae8ae8b67179c28f65c60bdbaebe0c28d21f7a996c66ae439cff Jul 06 19:34:15.026073 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:15.026038 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sfvcl" event={"ID":"5019d9af-5f01-470e-8a72-0a062617aa21","Type":"ContainerStarted","Data":"924b9871650eae8ae8b67179c28f65c60bdbaebe0c28d21f7a996c66ae439cff"} Jul 06 19:34:16.033416 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:16.033372 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sfvcl" event={"ID":"5019d9af-5f01-470e-8a72-0a062617aa21","Type":"ContainerStarted","Data":"56fb573610ba64dd826b6c0804fb30d19dfebf390c109620625cc4c5e15acd3f"} Jul 06 19:34:16.033727 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:16.033512 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sfvcl" Jul 06 19:34:16.058073 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:16.058021 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sfvcl" podStartSLOduration=1.5709550239999999 podStartE2EDuration="3.058003115s" totalImagesPullingTime="1.487048091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:34:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:34:16.054570483 +0000 UTC m=+464.874659221" watchObservedRunningTime="2026-07-06 19:34:16.058003115 +0000 UTC m=+464.878091827" Jul 06 19:34:18.809936 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:18.809900 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8j72"] Jul 06 19:34:18.818326 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:18.818304 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8j72" Jul 06 19:34:18.821105 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:18.821088 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-xdz2f\"" Jul 06 19:34:18.828547 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:18.828521 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8j72"] Jul 06 19:34:18.873725 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:18.873700 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7tgn\" (UniqueName: \"kubernetes.io/projected/dc654985-ff09-4530-a15f-0cfc01ea0b77-kube-api-access-g7tgn\") pod \"kuadrant-operator-controller-manager-8684c9ff49-m8j72\" (UID: \"dc654985-ff09-4530-a15f-0cfc01ea0b77\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8j72" Jul 06 19:34:18.873871 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:18.873845 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc654985-ff09-4530-a15f-0cfc01ea0b77-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-m8j72\" (UID: \"dc654985-ff09-4530-a15f-0cfc01ea0b77\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8j72" Jul 06 19:34:18.975466 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:18.975441 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7tgn\" (UniqueName: \"kubernetes.io/projected/dc654985-ff09-4530-a15f-0cfc01ea0b77-kube-api-access-g7tgn\") pod \"kuadrant-operator-controller-manager-8684c9ff49-m8j72\" (UID: \"dc654985-ff09-4530-a15f-0cfc01ea0b77\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8j72" Jul 06 19:34:18.975572 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:18.975493 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc654985-ff09-4530-a15f-0cfc01ea0b77-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-m8j72\" (UID: \"dc654985-ff09-4530-a15f-0cfc01ea0b77\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8j72" Jul 06 19:34:18.975783 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:18.975768 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc654985-ff09-4530-a15f-0cfc01ea0b77-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-m8j72\" (UID: \"dc654985-ff09-4530-a15f-0cfc01ea0b77\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8j72" Jul 06 19:34:18.995461 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:18.995437 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7tgn\" (UniqueName: \"kubernetes.io/projected/dc654985-ff09-4530-a15f-0cfc01ea0b77-kube-api-access-g7tgn\") pod \"kuadrant-operator-controller-manager-8684c9ff49-m8j72\" (UID: \"dc654985-ff09-4530-a15f-0cfc01ea0b77\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8j72" Jul 06 19:34:19.129197 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:19.129157 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8j72" Jul 06 19:34:19.270660 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:19.270634 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8j72"] Jul 06 19:34:19.271890 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:34:19.271864 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc654985_ff09_4530_a15f_0cfc01ea0b77.slice/crio-cace27d30642219e7b3b2e7507a730d1c3ebde7d746cc25d01320a42c34d8cec WatchSource:0}: Error finding container cace27d30642219e7b3b2e7507a730d1c3ebde7d746cc25d01320a42c34d8cec: Status 404 returned error can't find the container with id cace27d30642219e7b3b2e7507a730d1c3ebde7d746cc25d01320a42c34d8cec Jul 06 19:34:20.057405 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:20.057372 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8j72" event={"ID":"dc654985-ff09-4530-a15f-0cfc01ea0b77","Type":"ContainerStarted","Data":"cace27d30642219e7b3b2e7507a730d1c3ebde7d746cc25d01320a42c34d8cec"} Jul 06 19:34:22.859233 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:22.859204 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-5cc9j"] Jul 06 19:34:22.863151 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:22.863131 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5cc9j" Jul 06 19:34:22.865981 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:22.865961 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-jhnqp\"" Jul 06 19:34:22.878692 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:22.878668 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-5cc9j"] Jul 06 19:34:23.011529 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:23.011498 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbhlf\" (UniqueName: \"kubernetes.io/projected/ebc5be6f-3e75-4208-b275-a4752dbf5b06-kube-api-access-kbhlf\") pod \"authorino-operator-75cc8d5cb7-5cc9j\" (UID: \"ebc5be6f-3e75-4208-b275-a4752dbf5b06\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-5cc9j" Jul 06 19:34:23.112364 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:23.112279 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbhlf\" (UniqueName: \"kubernetes.io/projected/ebc5be6f-3e75-4208-b275-a4752dbf5b06-kube-api-access-kbhlf\") pod \"authorino-operator-75cc8d5cb7-5cc9j\" (UID: \"ebc5be6f-3e75-4208-b275-a4752dbf5b06\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-5cc9j" Jul 06 19:34:23.152788 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:23.152759 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbhlf\" (UniqueName: \"kubernetes.io/projected/ebc5be6f-3e75-4208-b275-a4752dbf5b06-kube-api-access-kbhlf\") pod \"authorino-operator-75cc8d5cb7-5cc9j\" (UID: \"ebc5be6f-3e75-4208-b275-a4752dbf5b06\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-5cc9j" Jul 06 19:34:23.176380 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:23.176355 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5cc9j" Jul 06 19:34:24.018153 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:24.018129 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8xjts" Jul 06 19:34:24.870266 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:24.870231 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-5cc9j"] Jul 06 19:34:24.871022 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:34:24.870998 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebc5be6f_3e75_4208_b275_a4752dbf5b06.slice/crio-40c09cb414bb4edfc23617b6b102c218b3b6dce7fe58c97ab4b18579b6e37d44 WatchSource:0}: Error finding container 40c09cb414bb4edfc23617b6b102c218b3b6dce7fe58c97ab4b18579b6e37d44: Status 404 returned error can't find the container with id 40c09cb414bb4edfc23617b6b102c218b3b6dce7fe58c97ab4b18579b6e37d44 Jul 06 19:34:25.093168 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:25.093133 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8j72" event={"ID":"dc654985-ff09-4530-a15f-0cfc01ea0b77","Type":"ContainerStarted","Data":"9d5c98d11afdf7ef0d5a4a778c75137265b24ec8ed2250989b662385725324fc"} Jul 06 19:34:25.093582 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:25.093346 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8j72" Jul 06 19:34:25.094462 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:25.094440 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5cc9j" event={"ID":"ebc5be6f-3e75-4208-b275-a4752dbf5b06","Type":"ContainerStarted","Data":"40c09cb414bb4edfc23617b6b102c218b3b6dce7fe58c97ab4b18579b6e37d44"} Jul 06 19:34:25.117994 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:25.117914 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8j72" podStartSLOduration=2.335240492 podStartE2EDuration="7.117901775s" totalImagesPullingTime="4.782661283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:34:25.116365963 +0000 UTC m=+473.936454673" watchObservedRunningTime="2026-07-06 19:34:25.117901775 +0000 UTC m=+473.937990487" Jul 06 19:34:27.041790 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:27.041756 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sfvcl" Jul 06 19:34:28.117253 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:28.117214 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5cc9j" event={"ID":"ebc5be6f-3e75-4208-b275-a4752dbf5b06","Type":"ContainerStarted","Data":"a30df086f307a4db8b7b25e9fc01dde496d7757508200ff08407de5687d77997"} Jul 06 19:34:28.117652 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:28.117333 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5cc9j" Jul 06 19:34:28.140385 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:28.140337 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5cc9j" podStartSLOduration=3.524505914 podStartE2EDuration="6.140321837s" totalImagesPullingTime="2.615815923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:34:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:34:28.139877146 +0000 UTC m=+476.959965848" watchObservedRunningTime="2026-07-06 19:34:28.140321837 +0000 UTC m=+476.960410548" Jul 06 19:34:36.104660 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:36.104633 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8j72" Jul 06 19:34:38.069886 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:38.069849 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8j72"] Jul 06 19:34:38.070350 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:38.070048 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8j72" podUID="dc654985-ff09-4530-a15f-0cfc01ea0b77" containerName="manager" containerID="cri-o://9d5c98d11afdf7ef0d5a4a778c75137265b24ec8ed2250989b662385725324fc" gracePeriod=2 Jul 06 19:34:38.080112 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:38.080087 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8j72"] Jul 06 19:34:38.107615 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:38.107583 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sfvcl"] Jul 06 19:34:38.107842 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:38.107796 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sfvcl" podUID="5019d9af-5f01-470e-8a72-0a062617aa21" containerName="manager" containerID="cri-o://56fb573610ba64dd826b6c0804fb30d19dfebf390c109620625cc4c5e15acd3f" gracePeriod=2 Jul 06 19:34:38.114232 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:34:38.114200 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-m8j72\" is forbidden: User \"system:node:ip-10-0-135-23.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-23.ec2.internal' and this object" podUID="dc654985-ff09-4530-a15f-0cfc01ea0b77" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8j72" Jul 06 19:34:38.118898 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:38.118866 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sfvcl"] Jul 06 19:34:38.135619 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:38.135597 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbtcd"] Jul 06 19:34:38.136000 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:38.135989 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5019d9af-5f01-470e-8a72-0a062617aa21" containerName="manager" Jul 06 19:34:38.136051 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:38.136003 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5019d9af-5f01-470e-8a72-0a062617aa21" containerName="manager" Jul 06 19:34:38.136051 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:38.136019 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc654985-ff09-4530-a15f-0cfc01ea0b77" containerName="manager" Jul 06 19:34:38.136051 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:38.136025 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc654985-ff09-4530-a15f-0cfc01ea0b77" containerName="manager" Jul 06 19:34:38.136140 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:38.136087 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc654985-ff09-4530-a15f-0cfc01ea0b77" containerName="manager" Jul 06 19:34:38.136140 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:38.136099 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5019d9af-5f01-470e-8a72-0a062617aa21" containerName="manager" Jul 06 19:34:38.139338 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:38.139317 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbtcd" Jul 06 19:34:38.142375 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:34:38.142346 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-m8j72\" is forbidden: User \"system:node:ip-10-0-135-23.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-23.ec2.internal' and this object" podUID="dc654985-ff09-4530-a15f-0cfc01ea0b77" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8j72" Jul 06 19:34:38.144657 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:34:38.144629 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-sfvcl\" is forbidden: User \"system:node:ip-10-0-135-23.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-23.ec2.internal' and this object" podUID="5019d9af-5f01-470e-8a72-0a062617aa21" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sfvcl" Jul 06 19:34:38.162015 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:38.161987 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x88hr"] Jul 06 19:34:38.165894 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:38.165864 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbtcd"] Jul 06 19:34:38.166003 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:38.165941 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x88hr" Jul 06 19:34:38.192987 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:38.192947 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x88hr"] Jul 06 19:34:38.206892 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:34:38.206856 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-m8j72\" is forbidden: User \"system:node:ip-10-0-135-23.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-23.ec2.internal' and this object" podUID="dc654985-ff09-4530-a15f-0cfc01ea0b77" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8j72" Jul 06 19:34:38.210374 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:34:38.210348 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-sfvcl\" is forbidden: User \"system:node:ip-10-0-135-23.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-23.ec2.internal' and this object" podUID="5019d9af-5f01-470e-8a72-0a062617aa21" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sfvcl" Jul 06 19:34:38.240509 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:38.240481 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4ed244e-88ad-4e23-9bda-9a157ee597ca-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zbtcd\" (UID: \"a4ed244e-88ad-4e23-9bda-9a157ee597ca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbtcd" Jul 06 19:34:38.240642 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:38.240549 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7dqx\" (UniqueName: \"kubernetes.io/projected/a4ed244e-88ad-4e23-9bda-9a157ee597ca-kube-api-access-c7dqx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zbtcd\" (UID: \"a4ed244e-88ad-4e23-9bda-9a157ee597ca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbtcd" Jul 06 19:34:38.341553 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:38.341521 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7dqx\" (UniqueName: \"kubernetes.io/projected/a4ed244e-88ad-4e23-9bda-9a157ee597ca-kube-api-access-c7dqx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zbtcd\" (UID: \"a4ed244e-88ad-4e23-9bda-9a157ee597ca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbtcd" Jul 06 19:34:38.341709 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:38.341691 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4ed244e-88ad-4e23-9bda-9a157ee597ca-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zbtcd\" (UID: \"a4ed244e-88ad-4e23-9bda-9a157ee597ca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbtcd" Jul 06 19:34:38.341796 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:38.341736 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxmz2\" (UniqueName: \"kubernetes.io/projected/3a81b584-f08e-4e8d-922c-5737522ef685-kube-api-access-lxmz2\") pod \"limitador-operator-controller-manager-859d7b78bd-x88hr\" (UID: \"3a81b584-f08e-4e8d-922c-5737522ef685\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x88hr" Jul 06 19:34:38.342133 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:38.342109 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4ed244e-88ad-4e23-9bda-9a157ee597ca-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zbtcd\" (UID: \"a4ed244e-88ad-4e23-9bda-9a157ee597ca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbtcd" Jul 06 19:34:38.359110 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:38.359088 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8j72" Jul 06 19:34:38.360725 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:38.360706 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7dqx\" (UniqueName: \"kubernetes.io/projected/a4ed244e-88ad-4e23-9bda-9a157ee597ca-kube-api-access-c7dqx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zbtcd\" (UID: \"a4ed244e-88ad-4e23-9bda-9a157ee597ca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbtcd" Jul 06 19:34:38.362266 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:38.362249 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sfvcl" Jul 06 19:34:38.363353 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:34:38.363329 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-sfvcl\" is forbidden: User \"system:node:ip-10-0-135-23.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-23.ec2.internal' and this object" podUID="5019d9af-5f01-470e-8a72-0a062617aa21" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sfvcl" Jul 06 19:34:38.366675 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:34:38.366652 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-m8j72\" is forbidden: User \"system:node:ip-10-0-135-23.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-23.ec2.internal' and this object" podUID="dc654985-ff09-4530-a15f-0cfc01ea0b77" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8j72" Jul 06 19:34:38.369025 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:34:38.369006 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-sfvcl\" is forbidden: User \"system:node:ip-10-0-135-23.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-23.ec2.internal' and this object" podUID="5019d9af-5f01-470e-8a72-0a062617aa21" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sfvcl" Jul 06 19:34:38.371110 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:34:38.371087 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-m8j72\" is forbidden: User \"system:node:ip-10-0-135-23.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-23.ec2.internal' and this object" podUID="dc654985-ff09-4530-a15f-0cfc01ea0b77" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8j72" Jul 06 19:34:38.442748 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:38.442726 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxmz2\" (UniqueName: \"kubernetes.io/projected/3a81b584-f08e-4e8d-922c-5737522ef685-kube-api-access-lxmz2\") pod \"limitador-operator-controller-manager-859d7b78bd-x88hr\" (UID: \"3a81b584-f08e-4e8d-922c-5737522ef685\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x88hr" Jul 06 19:34:38.460532 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:38.460509 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxmz2\" (UniqueName: \"kubernetes.io/projected/3a81b584-f08e-4e8d-922c-5737522ef685-kube-api-access-lxmz2\") pod \"limitador-operator-controller-manager-859d7b78bd-x88hr\" (UID: \"3a81b584-f08e-4e8d-922c-5737522ef685\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x88hr" Jul 06 19:34:38.502137 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:38.502112 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbtcd" Jul 06 19:34:38.508871 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:38.508849 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x88hr" Jul 06 19:34:38.543986 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:38.543959 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc654985-ff09-4530-a15f-0cfc01ea0b77-extensions-socket-volume\") pod \"dc654985-ff09-4530-a15f-0cfc01ea0b77\" (UID: \"dc654985-ff09-4530-a15f-0cfc01ea0b77\") " Jul 06 19:34:38.544126 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:38.544011 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r75zm\" (UniqueName: \"kubernetes.io/projected/5019d9af-5f01-470e-8a72-0a062617aa21-kube-api-access-r75zm\") pod \"5019d9af-5f01-470e-8a72-0a062617aa21\" (UID: \"5019d9af-5f01-470e-8a72-0a062617aa21\") " Jul 06 19:34:38.544126 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:38.544038 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7tgn\" (UniqueName: \"kubernetes.io/projected/dc654985-ff09-4530-a15f-0cfc01ea0b77-kube-api-access-g7tgn\") pod \"dc654985-ff09-4530-a15f-0cfc01ea0b77\" (UID: \"dc654985-ff09-4530-a15f-0cfc01ea0b77\") " Jul 06 19:34:38.544402 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:38.544374 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc654985-ff09-4530-a15f-0cfc01ea0b77-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "dc654985-ff09-4530-a15f-0cfc01ea0b77" (UID: "dc654985-ff09-4530-a15f-0cfc01ea0b77"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:34:38.546029 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:38.546001 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc654985-ff09-4530-a15f-0cfc01ea0b77-kube-api-access-g7tgn" (OuterVolumeSpecName: "kube-api-access-g7tgn") pod "dc654985-ff09-4530-a15f-0cfc01ea0b77" (UID: "dc654985-ff09-4530-a15f-0cfc01ea0b77"). InnerVolumeSpecName "kube-api-access-g7tgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:34:38.546265 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:38.546245 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5019d9af-5f01-470e-8a72-0a062617aa21-kube-api-access-r75zm" (OuterVolumeSpecName: "kube-api-access-r75zm") pod "5019d9af-5f01-470e-8a72-0a062617aa21" (UID: "5019d9af-5f01-470e-8a72-0a062617aa21"). InnerVolumeSpecName "kube-api-access-r75zm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:34:38.645778 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:38.645753 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc654985-ff09-4530-a15f-0cfc01ea0b77-extensions-socket-volume\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:34:38.645778 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:38.645777 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r75zm\" (UniqueName: \"kubernetes.io/projected/5019d9af-5f01-470e-8a72-0a062617aa21-kube-api-access-r75zm\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:34:38.645941 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:38.645788 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7tgn\" (UniqueName: \"kubernetes.io/projected/dc654985-ff09-4530-a15f-0cfc01ea0b77-kube-api-access-g7tgn\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:34:38.667859 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:38.667837 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbtcd"] Jul 06 19:34:38.669815 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:34:38.669792 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4ed244e_88ad_4e23_9bda_9a157ee597ca.slice/crio-33cc1f77f9c67133aa0a62601f09b6fd6be8c612b698ff65014fec1dc41c5639 WatchSource:0}: Error finding container 33cc1f77f9c67133aa0a62601f09b6fd6be8c612b698ff65014fec1dc41c5639: Status 404 returned error can't find the container with id 33cc1f77f9c67133aa0a62601f09b6fd6be8c612b698ff65014fec1dc41c5639 Jul 06 19:34:38.684875 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:38.684854 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x88hr"] Jul 06 19:34:38.688527 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:34:38.688502 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a81b584_f08e_4e8d_922c_5737522ef685.slice/crio-9eefa9528a8cd67304bf447154dba08006a2d486f132197b39d657e95ec90ffd WatchSource:0}: Error finding container 9eefa9528a8cd67304bf447154dba08006a2d486f132197b39d657e95ec90ffd: Status 404 returned error can't find the container with id 9eefa9528a8cd67304bf447154dba08006a2d486f132197b39d657e95ec90ffd Jul 06 19:34:39.125442 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:39.125420 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5cc9j" Jul 06 19:34:39.153495 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:34:39.153465 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-m8j72\" is forbidden: User \"system:node:ip-10-0-135-23.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-23.ec2.internal' and this object" podUID="dc654985-ff09-4530-a15f-0cfc01ea0b77" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8j72" Jul 06 19:34:39.159333 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:34:39.159308 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-sfvcl\" is forbidden: User \"system:node:ip-10-0-135-23.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-23.ec2.internal' and this object" podUID="5019d9af-5f01-470e-8a72-0a062617aa21" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sfvcl" Jul 06 19:34:39.183264 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:39.183223 2575 generic.go:356] "Generic (PLEG): container finished" podID="5019d9af-5f01-470e-8a72-0a062617aa21" containerID="56fb573610ba64dd826b6c0804fb30d19dfebf390c109620625cc4c5e15acd3f" exitCode=0 Jul 06 19:34:39.183264 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:39.183268 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sfvcl" Jul 06 19:34:39.183451 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:39.183333 2575 scope.go:117] "RemoveContainer" containerID="56fb573610ba64dd826b6c0804fb30d19dfebf390c109620625cc4c5e15acd3f" Jul 06 19:34:39.184860 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:39.184840 2575 generic.go:356] "Generic (PLEG): container finished" podID="dc654985-ff09-4530-a15f-0cfc01ea0b77" containerID="9d5c98d11afdf7ef0d5a4a778c75137265b24ec8ed2250989b662385725324fc" exitCode=0 Jul 06 19:34:39.184949 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:39.184889 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8j72" Jul 06 19:34:39.186680 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:39.186650 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x88hr" event={"ID":"3a81b584-f08e-4e8d-922c-5737522ef685","Type":"ContainerStarted","Data":"fae69ce0e23e71de2593c5fd187efc580b03a62e5117330f38eefeb64d58abf6"} Jul 06 19:34:39.186791 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:39.186697 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x88hr" event={"ID":"3a81b584-f08e-4e8d-922c-5737522ef685","Type":"ContainerStarted","Data":"9eefa9528a8cd67304bf447154dba08006a2d486f132197b39d657e95ec90ffd"} Jul 06 19:34:39.186791 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:39.186726 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x88hr" Jul 06 19:34:39.188521 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:39.188498 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbtcd" event={"ID":"a4ed244e-88ad-4e23-9bda-9a157ee597ca","Type":"ContainerStarted","Data":"e877b51af7476142ecf3946cee2a24b8ef2edf427e41463d9218b381e546778c"} Jul 06 19:34:39.188627 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:39.188540 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbtcd" event={"ID":"a4ed244e-88ad-4e23-9bda-9a157ee597ca","Type":"ContainerStarted","Data":"33cc1f77f9c67133aa0a62601f09b6fd6be8c612b698ff65014fec1dc41c5639"} Jul 06 19:34:39.188627 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:39.188598 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbtcd" Jul 06 19:34:39.193121 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:39.193103 2575 scope.go:117] "RemoveContainer" containerID="56fb573610ba64dd826b6c0804fb30d19dfebf390c109620625cc4c5e15acd3f" Jul 06 19:34:39.193434 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:34:39.193412 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56fb573610ba64dd826b6c0804fb30d19dfebf390c109620625cc4c5e15acd3f\": container with ID starting with 56fb573610ba64dd826b6c0804fb30d19dfebf390c109620625cc4c5e15acd3f not found: ID does not exist" containerID="56fb573610ba64dd826b6c0804fb30d19dfebf390c109620625cc4c5e15acd3f" Jul 06 19:34:39.193506 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:39.193447 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56fb573610ba64dd826b6c0804fb30d19dfebf390c109620625cc4c5e15acd3f"} err="failed to get container status \"56fb573610ba64dd826b6c0804fb30d19dfebf390c109620625cc4c5e15acd3f\": rpc error: code = NotFound desc = could not find container \"56fb573610ba64dd826b6c0804fb30d19dfebf390c109620625cc4c5e15acd3f\": container with ID starting with 56fb573610ba64dd826b6c0804fb30d19dfebf390c109620625cc4c5e15acd3f not found: ID does not exist" Jul 06 19:34:39.193506 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:39.193470 2575 scope.go:117] "RemoveContainer" containerID="9d5c98d11afdf7ef0d5a4a778c75137265b24ec8ed2250989b662385725324fc" Jul 06 19:34:39.203881 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:39.203859 2575 scope.go:117] "RemoveContainer" containerID="9d5c98d11afdf7ef0d5a4a778c75137265b24ec8ed2250989b662385725324fc" Jul 06 19:34:39.204289 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:34:39.204271 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d5c98d11afdf7ef0d5a4a778c75137265b24ec8ed2250989b662385725324fc\": container with ID starting with 9d5c98d11afdf7ef0d5a4a778c75137265b24ec8ed2250989b662385725324fc not found: ID does not exist" containerID="9d5c98d11afdf7ef0d5a4a778c75137265b24ec8ed2250989b662385725324fc" Jul 06 19:34:39.204366 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:39.204298 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d5c98d11afdf7ef0d5a4a778c75137265b24ec8ed2250989b662385725324fc"} err="failed to get container status \"9d5c98d11afdf7ef0d5a4a778c75137265b24ec8ed2250989b662385725324fc\": rpc error: code = NotFound desc = could not find container \"9d5c98d11afdf7ef0d5a4a778c75137265b24ec8ed2250989b662385725324fc\": container with ID starting with 9d5c98d11afdf7ef0d5a4a778c75137265b24ec8ed2250989b662385725324fc not found: ID does not exist" Jul 06 19:34:39.218679 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:34:39.218655 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-m8j72\" is forbidden: User \"system:node:ip-10-0-135-23.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-23.ec2.internal' and this object" podUID="dc654985-ff09-4530-a15f-0cfc01ea0b77" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8j72" Jul 06 19:34:39.221014 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:34:39.220996 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-sfvcl\" is forbidden: User \"system:node:ip-10-0-135-23.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-23.ec2.internal' and this object" podUID="5019d9af-5f01-470e-8a72-0a062617aa21" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sfvcl" Jul 06 19:34:39.266073 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:39.266019 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbtcd" podStartSLOduration=1.26600072 podStartE2EDuration="1.26600072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:34:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:34:39.264370272 +0000 UTC m=+488.084458984" watchObservedRunningTime="2026-07-06 19:34:39.26600072 +0000 UTC m=+488.086089433" Jul 06 19:34:39.299412 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:39.299369 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x88hr" podStartSLOduration=1.299355409 podStartE2EDuration="1.299355409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:34:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:34:39.295697205 +0000 UTC m=+488.115785939" watchObservedRunningTime="2026-07-06 19:34:39.299355409 +0000 UTC m=+488.119444119" Jul 06 19:34:39.795727 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:39.795695 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5019d9af-5f01-470e-8a72-0a062617aa21" path="/var/lib/kubelet/pods/5019d9af-5f01-470e-8a72-0a062617aa21/volumes" Jul 06 19:34:39.796034 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:39.796021 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc654985-ff09-4530-a15f-0cfc01ea0b77" path="/var/lib/kubelet/pods/dc654985-ff09-4530-a15f-0cfc01ea0b77/volumes" Jul 06 19:34:39.851191 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:39.851161 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gg299"] Jul 06 19:34:39.856431 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:39.856412 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gg299" Jul 06 19:34:39.876203 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:39.876166 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gg299"] Jul 06 19:34:39.957568 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:39.957542 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70853bf3-11eb-4d4b-8b27-e5be7b2dfe97-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-gg299\" (UID: \"70853bf3-11eb-4d4b-8b27-e5be7b2dfe97\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gg299" Jul 06 19:34:39.957677 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:39.957576 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gscfs\" (UniqueName: \"kubernetes.io/projected/70853bf3-11eb-4d4b-8b27-e5be7b2dfe97-kube-api-access-gscfs\") pod \"kuadrant-operator-controller-manager-86878c769f-gg299\" (UID: \"70853bf3-11eb-4d4b-8b27-e5be7b2dfe97\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gg299" Jul 06 19:34:40.058804 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:40.058742 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70853bf3-11eb-4d4b-8b27-e5be7b2dfe97-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-gg299\" (UID: \"70853bf3-11eb-4d4b-8b27-e5be7b2dfe97\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gg299" Jul 06 19:34:40.058804 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:40.058774 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gscfs\" (UniqueName: \"kubernetes.io/projected/70853bf3-11eb-4d4b-8b27-e5be7b2dfe97-kube-api-access-gscfs\") pod \"kuadrant-operator-controller-manager-86878c769f-gg299\" (UID: \"70853bf3-11eb-4d4b-8b27-e5be7b2dfe97\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gg299" Jul 06 19:34:40.059095 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:40.059079 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70853bf3-11eb-4d4b-8b27-e5be7b2dfe97-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-gg299\" (UID: \"70853bf3-11eb-4d4b-8b27-e5be7b2dfe97\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gg299" Jul 06 19:34:40.088836 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:40.088815 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gscfs\" (UniqueName: \"kubernetes.io/projected/70853bf3-11eb-4d4b-8b27-e5be7b2dfe97-kube-api-access-gscfs\") pod \"kuadrant-operator-controller-manager-86878c769f-gg299\" (UID: \"70853bf3-11eb-4d4b-8b27-e5be7b2dfe97\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gg299" Jul 06 19:34:40.166613 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:40.166584 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gg299" Jul 06 19:34:40.309818 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:40.309761 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gg299"] Jul 06 19:34:40.310798 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:34:40.310776 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70853bf3_11eb_4d4b_8b27_e5be7b2dfe97.slice/crio-935098efae8dbc218ff449c6a685f0bc40055dba351f1da757f8e5b68b3c01df WatchSource:0}: Error finding container 935098efae8dbc218ff449c6a685f0bc40055dba351f1da757f8e5b68b3c01df: Status 404 returned error can't find the container with id 935098efae8dbc218ff449c6a685f0bc40055dba351f1da757f8e5b68b3c01df Jul 06 19:34:41.205130 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:41.205093 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gg299" event={"ID":"70853bf3-11eb-4d4b-8b27-e5be7b2dfe97","Type":"ContainerStarted","Data":"869f462b48133f29b561b241eef491fca347a30bf1d839c8ea77791d0964b057"} Jul 06 19:34:41.205556 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:41.205172 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gg299" event={"ID":"70853bf3-11eb-4d4b-8b27-e5be7b2dfe97","Type":"ContainerStarted","Data":"935098efae8dbc218ff449c6a685f0bc40055dba351f1da757f8e5b68b3c01df"} Jul 06 19:34:41.205556 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:41.205207 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gg299" Jul 06 19:34:41.250570 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:41.250517 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gg299" podStartSLOduration=2.25050184 podStartE2EDuration="2.25050184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:34:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:34:41.246775299 +0000 UTC m=+490.066864010" watchObservedRunningTime="2026-07-06 19:34:41.25050184 +0000 UTC m=+490.070590550" Jul 06 19:34:50.199142 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:50.199113 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x88hr" Jul 06 19:34:50.199564 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:50.199215 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbtcd" Jul 06 19:34:52.214444 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:52.214412 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gg299" Jul 06 19:34:52.274438 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:52.274408 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbtcd"] Jul 06 19:34:52.274662 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:52.274635 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbtcd" podUID="a4ed244e-88ad-4e23-9bda-9a157ee597ca" containerName="manager" containerID="cri-o://e877b51af7476142ecf3946cee2a24b8ef2edf427e41463d9218b381e546778c" gracePeriod=10 Jul 06 19:34:52.522784 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:52.522762 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbtcd" Jul 06 19:34:52.552593 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:52.552568 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7dqx\" (UniqueName: \"kubernetes.io/projected/a4ed244e-88ad-4e23-9bda-9a157ee597ca-kube-api-access-c7dqx\") pod \"a4ed244e-88ad-4e23-9bda-9a157ee597ca\" (UID: \"a4ed244e-88ad-4e23-9bda-9a157ee597ca\") " Jul 06 19:34:52.552695 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:52.552617 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4ed244e-88ad-4e23-9bda-9a157ee597ca-extensions-socket-volume\") pod \"a4ed244e-88ad-4e23-9bda-9a157ee597ca\" (UID: \"a4ed244e-88ad-4e23-9bda-9a157ee597ca\") " Jul 06 19:34:52.553096 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:52.553064 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4ed244e-88ad-4e23-9bda-9a157ee597ca-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a4ed244e-88ad-4e23-9bda-9a157ee597ca" (UID: "a4ed244e-88ad-4e23-9bda-9a157ee597ca"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:34:52.554482 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:52.554465 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4ed244e-88ad-4e23-9bda-9a157ee597ca-kube-api-access-c7dqx" (OuterVolumeSpecName: "kube-api-access-c7dqx") pod "a4ed244e-88ad-4e23-9bda-9a157ee597ca" (UID: "a4ed244e-88ad-4e23-9bda-9a157ee597ca"). InnerVolumeSpecName "kube-api-access-c7dqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:34:52.653622 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:52.653595 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7dqx\" (UniqueName: \"kubernetes.io/projected/a4ed244e-88ad-4e23-9bda-9a157ee597ca-kube-api-access-c7dqx\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:34:52.653622 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:52.653616 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4ed244e-88ad-4e23-9bda-9a157ee597ca-extensions-socket-volume\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:34:53.280293 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:53.280261 2575 generic.go:356] "Generic (PLEG): container finished" podID="a4ed244e-88ad-4e23-9bda-9a157ee597ca" containerID="e877b51af7476142ecf3946cee2a24b8ef2edf427e41463d9218b381e546778c" exitCode=0 Jul 06 19:34:53.280672 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:53.280309 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbtcd" Jul 06 19:34:53.280672 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:53.280343 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbtcd" event={"ID":"a4ed244e-88ad-4e23-9bda-9a157ee597ca","Type":"ContainerDied","Data":"e877b51af7476142ecf3946cee2a24b8ef2edf427e41463d9218b381e546778c"} Jul 06 19:34:53.280672 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:53.280392 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbtcd" event={"ID":"a4ed244e-88ad-4e23-9bda-9a157ee597ca","Type":"ContainerDied","Data":"33cc1f77f9c67133aa0a62601f09b6fd6be8c612b698ff65014fec1dc41c5639"} Jul 06 19:34:53.280672 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:53.280414 2575 scope.go:117] "RemoveContainer" containerID="e877b51af7476142ecf3946cee2a24b8ef2edf427e41463d9218b381e546778c" Jul 06 19:34:53.290035 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:53.290017 2575 scope.go:117] "RemoveContainer" containerID="e877b51af7476142ecf3946cee2a24b8ef2edf427e41463d9218b381e546778c" Jul 06 19:34:53.290303 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:34:53.290285 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e877b51af7476142ecf3946cee2a24b8ef2edf427e41463d9218b381e546778c\": container with ID starting with e877b51af7476142ecf3946cee2a24b8ef2edf427e41463d9218b381e546778c not found: ID does not exist" containerID="e877b51af7476142ecf3946cee2a24b8ef2edf427e41463d9218b381e546778c" Jul 06 19:34:53.290351 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:53.290311 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e877b51af7476142ecf3946cee2a24b8ef2edf427e41463d9218b381e546778c"} err="failed to get container status \"e877b51af7476142ecf3946cee2a24b8ef2edf427e41463d9218b381e546778c\": rpc error: code = NotFound desc = could not find container \"e877b51af7476142ecf3946cee2a24b8ef2edf427e41463d9218b381e546778c\": container with ID starting with e877b51af7476142ecf3946cee2a24b8ef2edf427e41463d9218b381e546778c not found: ID does not exist" Jul 06 19:34:53.317308 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:53.317290 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbtcd"] Jul 06 19:34:53.336515 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:53.336488 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zbtcd"] Jul 06 19:34:53.794718 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:34:53.794688 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4ed244e-88ad-4e23-9bda-9a157ee597ca" path="/var/lib/kubelet/pods/a4ed244e-88ad-4e23-9bda-9a157ee597ca/volumes" Jul 06 19:35:10.528617 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:10.528552 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rwzcd"] Jul 06 19:35:10.529401 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:10.529372 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4ed244e-88ad-4e23-9bda-9a157ee597ca" containerName="manager" Jul 06 19:35:10.529401 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:10.529398 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4ed244e-88ad-4e23-9bda-9a157ee597ca" containerName="manager" Jul 06 19:35:10.529573 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:10.529497 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4ed244e-88ad-4e23-9bda-9a157ee597ca" containerName="manager" Jul 06 19:35:10.535955 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:10.535937 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-rwzcd" Jul 06 19:35:10.540362 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:10.540307 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 06 19:35:10.540666 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:10.540446 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-62dw5\"" Jul 06 19:35:10.544426 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:10.544405 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rwzcd"] Jul 06 19:35:10.599988 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:10.599955 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4a6a3527-cf9b-422f-85a3-64fa8722590e-config-file\") pod \"limitador-limitador-64c94fd64f-rwzcd\" (UID: \"4a6a3527-cf9b-422f-85a3-64fa8722590e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-rwzcd" Jul 06 19:35:10.599988 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:10.599985 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw2jn\" (UniqueName: \"kubernetes.io/projected/4a6a3527-cf9b-422f-85a3-64fa8722590e-kube-api-access-sw2jn\") pod \"limitador-limitador-64c94fd64f-rwzcd\" (UID: \"4a6a3527-cf9b-422f-85a3-64fa8722590e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-rwzcd" Jul 06 19:35:10.627216 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:10.627171 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rwzcd"] Jul 06 19:35:10.700685 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:10.700653 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4a6a3527-cf9b-422f-85a3-64fa8722590e-config-file\") pod \"limitador-limitador-64c94fd64f-rwzcd\" (UID: \"4a6a3527-cf9b-422f-85a3-64fa8722590e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-rwzcd" Jul 06 19:35:10.700835 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:10.700688 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw2jn\" (UniqueName: \"kubernetes.io/projected/4a6a3527-cf9b-422f-85a3-64fa8722590e-kube-api-access-sw2jn\") pod \"limitador-limitador-64c94fd64f-rwzcd\" (UID: \"4a6a3527-cf9b-422f-85a3-64fa8722590e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-rwzcd" Jul 06 19:35:10.701356 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:10.701338 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4a6a3527-cf9b-422f-85a3-64fa8722590e-config-file\") pod \"limitador-limitador-64c94fd64f-rwzcd\" (UID: \"4a6a3527-cf9b-422f-85a3-64fa8722590e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-rwzcd" Jul 06 19:35:10.716794 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:10.716768 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw2jn\" (UniqueName: \"kubernetes.io/projected/4a6a3527-cf9b-422f-85a3-64fa8722590e-kube-api-access-sw2jn\") pod \"limitador-limitador-64c94fd64f-rwzcd\" (UID: \"4a6a3527-cf9b-422f-85a3-64fa8722590e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-rwzcd" Jul 06 19:35:10.847269 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:10.847243 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-rwzcd" Jul 06 19:35:10.993259 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:10.993233 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rwzcd"] Jul 06 19:35:10.995434 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:35:10.995395 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a6a3527_cf9b_422f_85a3_64fa8722590e.slice/crio-a79986339196e4e86f99a04a3ddf808344c10f18145ddb3ba543baf7f1eb0633 WatchSource:0}: Error finding container a79986339196e4e86f99a04a3ddf808344c10f18145ddb3ba543baf7f1eb0633: Status 404 returned error can't find the container with id a79986339196e4e86f99a04a3ddf808344c10f18145ddb3ba543baf7f1eb0633 Jul 06 19:35:11.391290 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:11.391241 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-rwzcd" event={"ID":"4a6a3527-cf9b-422f-85a3-64fa8722590e","Type":"ContainerStarted","Data":"a79986339196e4e86f99a04a3ddf808344c10f18145ddb3ba543baf7f1eb0633"} Jul 06 19:35:11.475876 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:11.475782 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-xzq95"] Jul 06 19:35:11.480467 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:11.480439 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-xzq95" Jul 06 19:35:11.483413 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:11.483388 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-c5pcx\"" Jul 06 19:35:11.486678 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:11.486656 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-xzq95"] Jul 06 19:35:11.607735 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:11.607603 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg2s7\" (UniqueName: \"kubernetes.io/projected/d7ba38e5-7eee-4f48-a7cf-4c553ed312e2-kube-api-access-qg2s7\") pod \"authorino-58cd955675-xzq95\" (UID: \"d7ba38e5-7eee-4f48-a7cf-4c553ed312e2\") " pod="kuadrant-system/authorino-58cd955675-xzq95" Jul 06 19:35:11.708704 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:11.708628 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg2s7\" (UniqueName: \"kubernetes.io/projected/d7ba38e5-7eee-4f48-a7cf-4c553ed312e2-kube-api-access-qg2s7\") pod \"authorino-58cd955675-xzq95\" (UID: \"d7ba38e5-7eee-4f48-a7cf-4c553ed312e2\") " pod="kuadrant-system/authorino-58cd955675-xzq95" Jul 06 19:35:11.720172 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:11.720145 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg2s7\" (UniqueName: \"kubernetes.io/projected/d7ba38e5-7eee-4f48-a7cf-4c553ed312e2-kube-api-access-qg2s7\") pod \"authorino-58cd955675-xzq95\" (UID: \"d7ba38e5-7eee-4f48-a7cf-4c553ed312e2\") " pod="kuadrant-system/authorino-58cd955675-xzq95" Jul 06 19:35:11.791475 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:11.791294 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-xzq95" Jul 06 19:35:11.982673 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:11.982646 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-xzq95"] Jul 06 19:35:11.984353 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:35:11.984323 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7ba38e5_7eee_4f48_a7cf_4c553ed312e2.slice/crio-6e6cb365f804a4ba291c383197c7139a506130cf4d9ec107bd8acb3c80a1f396 WatchSource:0}: Error finding container 6e6cb365f804a4ba291c383197c7139a506130cf4d9ec107bd8acb3c80a1f396: Status 404 returned error can't find the container with id 6e6cb365f804a4ba291c383197c7139a506130cf4d9ec107bd8acb3c80a1f396 Jul 06 19:35:12.398249 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:12.398215 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-xzq95" event={"ID":"d7ba38e5-7eee-4f48-a7cf-4c553ed312e2","Type":"ContainerStarted","Data":"6e6cb365f804a4ba291c383197c7139a506130cf4d9ec107bd8acb3c80a1f396"} Jul 06 19:35:14.416582 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:14.416539 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-rwzcd" event={"ID":"4a6a3527-cf9b-422f-85a3-64fa8722590e","Type":"ContainerStarted","Data":"3e13f1ee37f54012ad1945ff04a4c7ec9c027f8bb891ec8cccd7beb70cf2ac54"} Jul 06 19:35:14.416992 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:14.416664 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-rwzcd" Jul 06 19:35:14.438384 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:14.438316 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-rwzcd" podStartSLOduration=1.749011015 podStartE2EDuration="4.438301566s" totalImagesPullingTime="2.689290551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:35:14.437689756 +0000 UTC m=+523.257778467" watchObservedRunningTime="2026-07-06 19:35:14.438301566 +0000 UTC m=+523.258390276" Jul 06 19:35:16.430411 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:16.430374 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-xzq95" event={"ID":"d7ba38e5-7eee-4f48-a7cf-4c553ed312e2","Type":"ContainerStarted","Data":"aa5f2a89aaebc02bcdfeeeb51d6e03fa7f9053299916269c125b709a79d042a4"} Jul 06 19:35:16.449446 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:16.449278 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-xzq95" podStartSLOduration=1.64362893 podStartE2EDuration="5.44926175s" totalImagesPullingTime="3.80563282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:35:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:35:16.448582368 +0000 UTC m=+525.268671079" watchObservedRunningTime="2026-07-06 19:35:16.44926175 +0000 UTC m=+525.269350461" Jul 06 19:35:16.732339 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:16.732263 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-xzq95"] Jul 06 19:35:18.442596 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:18.442563 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-xzq95" podUID="d7ba38e5-7eee-4f48-a7cf-4c553ed312e2" containerName="authorino" containerID="cri-o://aa5f2a89aaebc02bcdfeeeb51d6e03fa7f9053299916269c125b709a79d042a4" gracePeriod=30 Jul 06 19:35:18.690471 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:18.690451 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-xzq95" Jul 06 19:35:18.774507 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:18.774452 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qg2s7\" (UniqueName: \"kubernetes.io/projected/d7ba38e5-7eee-4f48-a7cf-4c553ed312e2-kube-api-access-qg2s7\") pod \"d7ba38e5-7eee-4f48-a7cf-4c553ed312e2\" (UID: \"d7ba38e5-7eee-4f48-a7cf-4c553ed312e2\") " Jul 06 19:35:18.776350 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:18.776321 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7ba38e5-7eee-4f48-a7cf-4c553ed312e2-kube-api-access-qg2s7" (OuterVolumeSpecName: "kube-api-access-qg2s7") pod "d7ba38e5-7eee-4f48-a7cf-4c553ed312e2" (UID: "d7ba38e5-7eee-4f48-a7cf-4c553ed312e2"). InnerVolumeSpecName "kube-api-access-qg2s7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:35:18.875715 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:18.875691 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg2s7\" (UniqueName: \"kubernetes.io/projected/d7ba38e5-7eee-4f48-a7cf-4c553ed312e2-kube-api-access-qg2s7\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:35:19.448993 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:19.448960 2575 generic.go:356] "Generic (PLEG): container finished" podID="d7ba38e5-7eee-4f48-a7cf-4c553ed312e2" containerID="aa5f2a89aaebc02bcdfeeeb51d6e03fa7f9053299916269c125b709a79d042a4" exitCode=0 Jul 06 19:35:19.449389 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:19.449007 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-xzq95" Jul 06 19:35:19.449389 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:19.449042 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-xzq95" event={"ID":"d7ba38e5-7eee-4f48-a7cf-4c553ed312e2","Type":"ContainerDied","Data":"aa5f2a89aaebc02bcdfeeeb51d6e03fa7f9053299916269c125b709a79d042a4"} Jul 06 19:35:19.449389 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:19.449081 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-xzq95" event={"ID":"d7ba38e5-7eee-4f48-a7cf-4c553ed312e2","Type":"ContainerDied","Data":"6e6cb365f804a4ba291c383197c7139a506130cf4d9ec107bd8acb3c80a1f396"} Jul 06 19:35:19.449389 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:19.449097 2575 scope.go:117] "RemoveContainer" containerID="aa5f2a89aaebc02bcdfeeeb51d6e03fa7f9053299916269c125b709a79d042a4" Jul 06 19:35:19.458658 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:19.458641 2575 scope.go:117] "RemoveContainer" containerID="aa5f2a89aaebc02bcdfeeeb51d6e03fa7f9053299916269c125b709a79d042a4" Jul 06 19:35:19.458889 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:35:19.458872 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa5f2a89aaebc02bcdfeeeb51d6e03fa7f9053299916269c125b709a79d042a4\": container with ID starting with aa5f2a89aaebc02bcdfeeeb51d6e03fa7f9053299916269c125b709a79d042a4 not found: ID does not exist" containerID="aa5f2a89aaebc02bcdfeeeb51d6e03fa7f9053299916269c125b709a79d042a4" Jul 06 19:35:19.458933 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:19.458897 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa5f2a89aaebc02bcdfeeeb51d6e03fa7f9053299916269c125b709a79d042a4"} err="failed to get container status \"aa5f2a89aaebc02bcdfeeeb51d6e03fa7f9053299916269c125b709a79d042a4\": rpc error: code = NotFound desc = could not find container \"aa5f2a89aaebc02bcdfeeeb51d6e03fa7f9053299916269c125b709a79d042a4\": container with ID starting with aa5f2a89aaebc02bcdfeeeb51d6e03fa7f9053299916269c125b709a79d042a4 not found: ID does not exist" Jul 06 19:35:19.473729 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:19.473708 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-xzq95"] Jul 06 19:35:19.478989 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:19.478971 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-xzq95"] Jul 06 19:35:19.794725 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:19.794659 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7ba38e5-7eee-4f48-a7cf-4c553ed312e2" path="/var/lib/kubelet/pods/d7ba38e5-7eee-4f48-a7cf-4c553ed312e2/volumes" Jul 06 19:35:25.424503 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:25.424436 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-rwzcd" Jul 06 19:35:25.723486 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:25.723408 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rwzcd"] Jul 06 19:35:25.723636 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:25.723571 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-rwzcd" podUID="4a6a3527-cf9b-422f-85a3-64fa8722590e" containerName="limitador" containerID="cri-o://3e13f1ee37f54012ad1945ff04a4c7ec9c027f8bb891ec8cccd7beb70cf2ac54" gracePeriod=30 Jul 06 19:35:26.274826 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:26.274804 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-rwzcd" Jul 06 19:35:26.433273 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:26.433249 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sw2jn\" (UniqueName: \"kubernetes.io/projected/4a6a3527-cf9b-422f-85a3-64fa8722590e-kube-api-access-sw2jn\") pod \"4a6a3527-cf9b-422f-85a3-64fa8722590e\" (UID: \"4a6a3527-cf9b-422f-85a3-64fa8722590e\") " Jul 06 19:35:26.433592 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:26.433288 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4a6a3527-cf9b-422f-85a3-64fa8722590e-config-file\") pod \"4a6a3527-cf9b-422f-85a3-64fa8722590e\" (UID: \"4a6a3527-cf9b-422f-85a3-64fa8722590e\") " Jul 06 19:35:26.433644 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:26.433623 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a6a3527-cf9b-422f-85a3-64fa8722590e-config-file" (OuterVolumeSpecName: "config-file") pod "4a6a3527-cf9b-422f-85a3-64fa8722590e" (UID: "4a6a3527-cf9b-422f-85a3-64fa8722590e"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:35:26.435080 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:26.435051 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a6a3527-cf9b-422f-85a3-64fa8722590e-kube-api-access-sw2jn" (OuterVolumeSpecName: "kube-api-access-sw2jn") pod "4a6a3527-cf9b-422f-85a3-64fa8722590e" (UID: "4a6a3527-cf9b-422f-85a3-64fa8722590e"). InnerVolumeSpecName "kube-api-access-sw2jn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:35:26.490601 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:26.490577 2575 generic.go:356] "Generic (PLEG): container finished" podID="4a6a3527-cf9b-422f-85a3-64fa8722590e" containerID="3e13f1ee37f54012ad1945ff04a4c7ec9c027f8bb891ec8cccd7beb70cf2ac54" exitCode=0 Jul 06 19:35:26.490699 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:26.490616 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-rwzcd" Jul 06 19:35:26.490699 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:26.490651 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-rwzcd" event={"ID":"4a6a3527-cf9b-422f-85a3-64fa8722590e","Type":"ContainerDied","Data":"3e13f1ee37f54012ad1945ff04a4c7ec9c027f8bb891ec8cccd7beb70cf2ac54"} Jul 06 19:35:26.490770 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:26.490702 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-rwzcd" event={"ID":"4a6a3527-cf9b-422f-85a3-64fa8722590e","Type":"ContainerDied","Data":"a79986339196e4e86f99a04a3ddf808344c10f18145ddb3ba543baf7f1eb0633"} Jul 06 19:35:26.490770 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:26.490724 2575 scope.go:117] "RemoveContainer" containerID="3e13f1ee37f54012ad1945ff04a4c7ec9c027f8bb891ec8cccd7beb70cf2ac54" Jul 06 19:35:26.499951 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:26.499936 2575 scope.go:117] "RemoveContainer" containerID="3e13f1ee37f54012ad1945ff04a4c7ec9c027f8bb891ec8cccd7beb70cf2ac54" Jul 06 19:35:26.500230 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:35:26.500202 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e13f1ee37f54012ad1945ff04a4c7ec9c027f8bb891ec8cccd7beb70cf2ac54\": container with ID starting with 3e13f1ee37f54012ad1945ff04a4c7ec9c027f8bb891ec8cccd7beb70cf2ac54 not found: ID does not exist" containerID="3e13f1ee37f54012ad1945ff04a4c7ec9c027f8bb891ec8cccd7beb70cf2ac54" Jul 06 19:35:26.500324 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:26.500237 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e13f1ee37f54012ad1945ff04a4c7ec9c027f8bb891ec8cccd7beb70cf2ac54"} err="failed to get container status \"3e13f1ee37f54012ad1945ff04a4c7ec9c027f8bb891ec8cccd7beb70cf2ac54\": rpc error: code = NotFound desc = could not find container \"3e13f1ee37f54012ad1945ff04a4c7ec9c027f8bb891ec8cccd7beb70cf2ac54\": container with ID starting with 3e13f1ee37f54012ad1945ff04a4c7ec9c027f8bb891ec8cccd7beb70cf2ac54 not found: ID does not exist" Jul 06 19:35:26.520220 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:26.520199 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rwzcd"] Jul 06 19:35:26.523432 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:26.523412 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rwzcd"] Jul 06 19:35:26.534524 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:26.534505 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw2jn\" (UniqueName: \"kubernetes.io/projected/4a6a3527-cf9b-422f-85a3-64fa8722590e-kube-api-access-sw2jn\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:35:26.534596 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:26.534526 2575 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4a6a3527-cf9b-422f-85a3-64fa8722590e-config-file\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:35:27.794679 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:27.794643 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a6a3527-cf9b-422f-85a3-64fa8722590e" path="/var/lib/kubelet/pods/4a6a3527-cf9b-422f-85a3-64fa8722590e/volumes" Jul 06 19:35:32.177244 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:32.177215 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/postgres-69fb4594cf-jk6zp"] Jul 06 19:35:32.178274 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:32.178255 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7ba38e5-7eee-4f48-a7cf-4c553ed312e2" containerName="authorino" Jul 06 19:35:32.178274 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:32.178276 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7ba38e5-7eee-4f48-a7cf-4c553ed312e2" containerName="authorino" Jul 06 19:35:32.178451 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:32.178289 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a6a3527-cf9b-422f-85a3-64fa8722590e" containerName="limitador" Jul 06 19:35:32.178451 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:32.178298 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a6a3527-cf9b-422f-85a3-64fa8722590e" containerName="limitador" Jul 06 19:35:32.178451 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:32.178404 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7ba38e5-7eee-4f48-a7cf-4c553ed312e2" containerName="authorino" Jul 06 19:35:32.178451 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:32.178416 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a6a3527-cf9b-422f-85a3-64fa8722590e" containerName="limitador" Jul 06 19:35:32.182973 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:32.182955 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-jk6zp" Jul 06 19:35:32.186700 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:32.186674 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"postgres-creds\"" Jul 06 19:35:32.186828 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:32.186700 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zcf87\"" Jul 06 19:35:32.193715 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:32.193688 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-jk6zp"] Jul 06 19:35:32.277293 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:32.277267 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn472\" (UniqueName: \"kubernetes.io/projected/05eddb0a-97b3-4aca-8d15-a1f739f6d45d-kube-api-access-zn472\") pod \"postgres-69fb4594cf-jk6zp\" (UID: \"05eddb0a-97b3-4aca-8d15-a1f739f6d45d\") " pod="opendatahub/postgres-69fb4594cf-jk6zp" Jul 06 19:35:32.277412 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:32.277314 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/05eddb0a-97b3-4aca-8d15-a1f739f6d45d-data\") pod \"postgres-69fb4594cf-jk6zp\" (UID: \"05eddb0a-97b3-4aca-8d15-a1f739f6d45d\") " pod="opendatahub/postgres-69fb4594cf-jk6zp" Jul 06 19:35:32.377942 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:32.377917 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn472\" (UniqueName: \"kubernetes.io/projected/05eddb0a-97b3-4aca-8d15-a1f739f6d45d-kube-api-access-zn472\") pod \"postgres-69fb4594cf-jk6zp\" (UID: \"05eddb0a-97b3-4aca-8d15-a1f739f6d45d\") " pod="opendatahub/postgres-69fb4594cf-jk6zp" Jul 06 19:35:32.378043 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:32.377960 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/05eddb0a-97b3-4aca-8d15-a1f739f6d45d-data\") pod \"postgres-69fb4594cf-jk6zp\" (UID: \"05eddb0a-97b3-4aca-8d15-a1f739f6d45d\") " pod="opendatahub/postgres-69fb4594cf-jk6zp" Jul 06 19:35:32.378313 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:32.378298 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/05eddb0a-97b3-4aca-8d15-a1f739f6d45d-data\") pod \"postgres-69fb4594cf-jk6zp\" (UID: \"05eddb0a-97b3-4aca-8d15-a1f739f6d45d\") " pod="opendatahub/postgres-69fb4594cf-jk6zp" Jul 06 19:35:32.388849 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:32.388828 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn472\" (UniqueName: \"kubernetes.io/projected/05eddb0a-97b3-4aca-8d15-a1f739f6d45d-kube-api-access-zn472\") pod \"postgres-69fb4594cf-jk6zp\" (UID: \"05eddb0a-97b3-4aca-8d15-a1f739f6d45d\") " pod="opendatahub/postgres-69fb4594cf-jk6zp" Jul 06 19:35:32.504781 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:32.504719 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-jk6zp" Jul 06 19:35:32.641660 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:32.641636 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-jk6zp"] Jul 06 19:35:32.642618 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:35:32.642590 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05eddb0a_97b3_4aca_8d15_a1f739f6d45d.slice/crio-788b6722debe36fd50f273855f1188035f69414794c8ad978249349f2b40fcda WatchSource:0}: Error finding container 788b6722debe36fd50f273855f1188035f69414794c8ad978249349f2b40fcda: Status 404 returned error can't find the container with id 788b6722debe36fd50f273855f1188035f69414794c8ad978249349f2b40fcda Jul 06 19:35:33.536702 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:33.536658 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-jk6zp" event={"ID":"05eddb0a-97b3-4aca-8d15-a1f739f6d45d","Type":"ContainerStarted","Data":"788b6722debe36fd50f273855f1188035f69414794c8ad978249349f2b40fcda"} Jul 06 19:35:37.564417 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:37.564385 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-jk6zp" event={"ID":"05eddb0a-97b3-4aca-8d15-a1f739f6d45d","Type":"ContainerStarted","Data":"ee4b35bfd1e9795987c14a27cc070872567b00ff138f4bed005553101b9b2042"} Jul 06 19:35:37.564791 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:37.564526 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/postgres-69fb4594cf-jk6zp" Jul 06 19:35:37.595629 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:37.595587 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/postgres-69fb4594cf-jk6zp" podStartSLOduration=1.2241923080000001 podStartE2EDuration="5.59557541s" totalImagesPullingTime="4.371383102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:35:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:35:37.593645196 +0000 UTC m=+546.413733932" watchObservedRunningTime="2026-07-06 19:35:37.59557541 +0000 UTC m=+546.415664121" Jul 06 19:35:43.607147 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:43.607122 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/postgres-69fb4594cf-jk6zp" Jul 06 19:35:44.188139 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:44.188099 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jf7ks"] Jul 06 19:35:44.195506 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:44.195483 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-jf7ks" Jul 06 19:35:44.198168 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:44.198147 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-c5pcx\"" Jul 06 19:35:44.200024 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:44.200003 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jf7ks"] Jul 06 19:35:44.376375 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:44.376346 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nm4p\" (UniqueName: \"kubernetes.io/projected/fee663a5-0cba-4040-b630-719f998362cb-kube-api-access-9nm4p\") pod \"authorino-5d4b67c446-jf7ks\" (UID: \"fee663a5-0cba-4040-b630-719f998362cb\") " pod="kuadrant-system/authorino-5d4b67c446-jf7ks" Jul 06 19:35:44.445106 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:44.445043 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jf7ks"] Jul 06 19:35:44.445309 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:35:44.445288 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-9nm4p], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-jf7ks" podUID="fee663a5-0cba-4040-b630-719f998362cb" Jul 06 19:35:44.477685 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:44.477660 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nm4p\" (UniqueName: \"kubernetes.io/projected/fee663a5-0cba-4040-b630-719f998362cb-kube-api-access-9nm4p\") pod \"authorino-5d4b67c446-jf7ks\" (UID: \"fee663a5-0cba-4040-b630-719f998362cb\") " pod="kuadrant-system/authorino-5d4b67c446-jf7ks" Jul 06 19:35:44.503745 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:44.503722 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nm4p\" (UniqueName: \"kubernetes.io/projected/fee663a5-0cba-4040-b630-719f998362cb-kube-api-access-9nm4p\") pod \"authorino-5d4b67c446-jf7ks\" (UID: \"fee663a5-0cba-4040-b630-719f998362cb\") " pod="kuadrant-system/authorino-5d4b67c446-jf7ks" Jul 06 19:35:44.614716 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:44.614693 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-jf7ks" Jul 06 19:35:44.639995 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:44.639975 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-jf7ks" Jul 06 19:35:44.651837 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:44.651810 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6ffb7848c6-qjd89"] Jul 06 19:35:44.660977 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:44.660959 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6ffb7848c6-qjd89" Jul 06 19:35:44.664050 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:44.663975 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 06 19:35:44.667136 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:44.667115 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6ffb7848c6-qjd89"] Jul 06 19:35:44.779898 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:44.779840 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9nm4p\" (UniqueName: \"kubernetes.io/projected/fee663a5-0cba-4040-b630-719f998362cb-kube-api-access-9nm4p\") pod \"fee663a5-0cba-4040-b630-719f998362cb\" (UID: \"fee663a5-0cba-4040-b630-719f998362cb\") " Jul 06 19:35:44.779995 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:44.779955 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/67a17400-c96e-4ed6-9c7a-bfe7c5a8ee49-tls-cert\") pod \"authorino-6ffb7848c6-qjd89\" (UID: \"67a17400-c96e-4ed6-9c7a-bfe7c5a8ee49\") " pod="kuadrant-system/authorino-6ffb7848c6-qjd89" Jul 06 19:35:44.780038 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:44.780007 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shntg\" (UniqueName: \"kubernetes.io/projected/67a17400-c96e-4ed6-9c7a-bfe7c5a8ee49-kube-api-access-shntg\") pod \"authorino-6ffb7848c6-qjd89\" (UID: \"67a17400-c96e-4ed6-9c7a-bfe7c5a8ee49\") " pod="kuadrant-system/authorino-6ffb7848c6-qjd89" Jul 06 19:35:44.781825 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:44.781803 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fee663a5-0cba-4040-b630-719f998362cb-kube-api-access-9nm4p" (OuterVolumeSpecName: "kube-api-access-9nm4p") pod "fee663a5-0cba-4040-b630-719f998362cb" (UID: "fee663a5-0cba-4040-b630-719f998362cb"). InnerVolumeSpecName "kube-api-access-9nm4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:35:44.881123 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:44.881096 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/67a17400-c96e-4ed6-9c7a-bfe7c5a8ee49-tls-cert\") pod \"authorino-6ffb7848c6-qjd89\" (UID: \"67a17400-c96e-4ed6-9c7a-bfe7c5a8ee49\") " pod="kuadrant-system/authorino-6ffb7848c6-qjd89" Jul 06 19:35:44.881228 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:44.881145 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shntg\" (UniqueName: \"kubernetes.io/projected/67a17400-c96e-4ed6-9c7a-bfe7c5a8ee49-kube-api-access-shntg\") pod \"authorino-6ffb7848c6-qjd89\" (UID: \"67a17400-c96e-4ed6-9c7a-bfe7c5a8ee49\") " pod="kuadrant-system/authorino-6ffb7848c6-qjd89" Jul 06 19:35:44.881228 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:44.881208 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nm4p\" (UniqueName: \"kubernetes.io/projected/fee663a5-0cba-4040-b630-719f998362cb-kube-api-access-9nm4p\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:35:44.883335 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:44.883314 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/67a17400-c96e-4ed6-9c7a-bfe7c5a8ee49-tls-cert\") pod \"authorino-6ffb7848c6-qjd89\" (UID: \"67a17400-c96e-4ed6-9c7a-bfe7c5a8ee49\") " pod="kuadrant-system/authorino-6ffb7848c6-qjd89" Jul 06 19:35:44.892148 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:44.892131 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shntg\" (UniqueName: \"kubernetes.io/projected/67a17400-c96e-4ed6-9c7a-bfe7c5a8ee49-kube-api-access-shntg\") pod \"authorino-6ffb7848c6-qjd89\" (UID: \"67a17400-c96e-4ed6-9c7a-bfe7c5a8ee49\") " pod="kuadrant-system/authorino-6ffb7848c6-qjd89" Jul 06 19:35:44.970624 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:44.970606 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6ffb7848c6-qjd89" Jul 06 19:35:45.143279 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:45.143256 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6ffb7848c6-qjd89"] Jul 06 19:35:45.144834 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:35:45.144801 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67a17400_c96e_4ed6_9c7a_bfe7c5a8ee49.slice/crio-51c7b015d7e7fadc5e8e2438ed80fc60c7e4c7ec66b930878e6700ca75008389 WatchSource:0}: Error finding container 51c7b015d7e7fadc5e8e2438ed80fc60c7e4c7ec66b930878e6700ca75008389: Status 404 returned error can't find the container with id 51c7b015d7e7fadc5e8e2438ed80fc60c7e4c7ec66b930878e6700ca75008389 Jul 06 19:35:45.621716 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:45.621680 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6ffb7848c6-qjd89" event={"ID":"67a17400-c96e-4ed6-9c7a-bfe7c5a8ee49","Type":"ContainerStarted","Data":"51c7b015d7e7fadc5e8e2438ed80fc60c7e4c7ec66b930878e6700ca75008389"} Jul 06 19:35:45.622204 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:45.621735 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-jf7ks" Jul 06 19:35:45.698261 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:45.698242 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jf7ks"] Jul 06 19:35:45.702230 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:45.702207 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jf7ks"] Jul 06 19:35:45.795600 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:45.795574 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fee663a5-0cba-4040-b630-719f998362cb" path="/var/lib/kubelet/pods/fee663a5-0cba-4040-b630-719f998362cb/volumes" Jul 06 19:35:46.629352 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:46.629315 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6ffb7848c6-qjd89" event={"ID":"67a17400-c96e-4ed6-9c7a-bfe7c5a8ee49","Type":"ContainerStarted","Data":"5a198627b47e6fdf90d34ea59991255d3453a6ba8004b5657bbc6fd547088ba3"} Jul 06 19:35:46.648107 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:46.648066 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6ffb7848c6-qjd89" podStartSLOduration=2.158839297 podStartE2EDuration="2.648051414s" totalImagesPullingTime="489.212117ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:35:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:35:46.647198453 +0000 UTC m=+555.467287159" watchObservedRunningTime="2026-07-06 19:35:46.648051414 +0000 UTC m=+555.468140125" Jul 06 19:35:51.177829 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:51.177797 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-57b67684d-5s9gj"] Jul 06 19:35:51.193685 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:51.193659 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57b67684d-5s9gj" Jul 06 19:35:51.193861 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:51.193662 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57b67684d-5s9gj"] Jul 06 19:35:51.196427 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:51.196400 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 06 19:35:51.196427 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:51.196421 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nwfhq\"" Jul 06 19:35:51.196615 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:51.196428 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 06 19:35:51.235989 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:51.235962 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/83b666c6-a8c0-482d-8db0-f596b3cf70de-webhook-cert\") pod \"maas-controller-57b67684d-5s9gj\" (UID: \"83b666c6-a8c0-482d-8db0-f596b3cf70de\") " pod="opendatahub/maas-controller-57b67684d-5s9gj" Jul 06 19:35:51.236125 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:51.236047 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzqd8\" (UniqueName: \"kubernetes.io/projected/83b666c6-a8c0-482d-8db0-f596b3cf70de-kube-api-access-hzqd8\") pod \"maas-controller-57b67684d-5s9gj\" (UID: \"83b666c6-a8c0-482d-8db0-f596b3cf70de\") " pod="opendatahub/maas-controller-57b67684d-5s9gj" Jul 06 19:35:51.336909 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:51.336882 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/83b666c6-a8c0-482d-8db0-f596b3cf70de-webhook-cert\") pod \"maas-controller-57b67684d-5s9gj\" (UID: \"83b666c6-a8c0-482d-8db0-f596b3cf70de\") " pod="opendatahub/maas-controller-57b67684d-5s9gj" Jul 06 19:35:51.337039 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:51.336937 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzqd8\" (UniqueName: \"kubernetes.io/projected/83b666c6-a8c0-482d-8db0-f596b3cf70de-kube-api-access-hzqd8\") pod \"maas-controller-57b67684d-5s9gj\" (UID: \"83b666c6-a8c0-482d-8db0-f596b3cf70de\") " pod="opendatahub/maas-controller-57b67684d-5s9gj" Jul 06 19:35:51.339227 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:51.339208 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/83b666c6-a8c0-482d-8db0-f596b3cf70de-webhook-cert\") pod \"maas-controller-57b67684d-5s9gj\" (UID: \"83b666c6-a8c0-482d-8db0-f596b3cf70de\") " pod="opendatahub/maas-controller-57b67684d-5s9gj" Jul 06 19:35:51.352132 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:51.352106 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzqd8\" (UniqueName: \"kubernetes.io/projected/83b666c6-a8c0-482d-8db0-f596b3cf70de-kube-api-access-hzqd8\") pod \"maas-controller-57b67684d-5s9gj\" (UID: \"83b666c6-a8c0-482d-8db0-f596b3cf70de\") " pod="opendatahub/maas-controller-57b67684d-5s9gj" Jul 06 19:35:51.381074 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:51.381050 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-89b4bf89b-dthvn"] Jul 06 19:35:51.383919 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:51.383904 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-89b4bf89b-dthvn" Jul 06 19:35:51.395992 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:51.395973 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-89b4bf89b-dthvn"] Jul 06 19:35:51.437730 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:51.437679 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1dc9ce5e-0be5-4f33-8d82-70b915ba1caa-webhook-cert\") pod \"maas-controller-89b4bf89b-dthvn\" (UID: \"1dc9ce5e-0be5-4f33-8d82-70b915ba1caa\") " pod="opendatahub/maas-controller-89b4bf89b-dthvn" Jul 06 19:35:51.437730 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:51.437721 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn8zh\" (UniqueName: \"kubernetes.io/projected/1dc9ce5e-0be5-4f33-8d82-70b915ba1caa-kube-api-access-pn8zh\") pod \"maas-controller-89b4bf89b-dthvn\" (UID: \"1dc9ce5e-0be5-4f33-8d82-70b915ba1caa\") " pod="opendatahub/maas-controller-89b4bf89b-dthvn" Jul 06 19:35:51.505586 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:51.505564 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57b67684d-5s9gj" Jul 06 19:35:51.538410 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:51.538384 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1dc9ce5e-0be5-4f33-8d82-70b915ba1caa-webhook-cert\") pod \"maas-controller-89b4bf89b-dthvn\" (UID: \"1dc9ce5e-0be5-4f33-8d82-70b915ba1caa\") " pod="opendatahub/maas-controller-89b4bf89b-dthvn" Jul 06 19:35:51.538525 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:51.538444 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn8zh\" (UniqueName: \"kubernetes.io/projected/1dc9ce5e-0be5-4f33-8d82-70b915ba1caa-kube-api-access-pn8zh\") pod \"maas-controller-89b4bf89b-dthvn\" (UID: \"1dc9ce5e-0be5-4f33-8d82-70b915ba1caa\") " pod="opendatahub/maas-controller-89b4bf89b-dthvn" Jul 06 19:35:51.540904 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:51.540883 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1dc9ce5e-0be5-4f33-8d82-70b915ba1caa-webhook-cert\") pod \"maas-controller-89b4bf89b-dthvn\" (UID: \"1dc9ce5e-0be5-4f33-8d82-70b915ba1caa\") " pod="opendatahub/maas-controller-89b4bf89b-dthvn" Jul 06 19:35:51.548504 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:51.548484 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn8zh\" (UniqueName: \"kubernetes.io/projected/1dc9ce5e-0be5-4f33-8d82-70b915ba1caa-kube-api-access-pn8zh\") pod \"maas-controller-89b4bf89b-dthvn\" (UID: \"1dc9ce5e-0be5-4f33-8d82-70b915ba1caa\") " pod="opendatahub/maas-controller-89b4bf89b-dthvn" Jul 06 19:35:51.642618 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:51.642541 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57b67684d-5s9gj"] Jul 06 19:35:51.644948 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:35:51.644920 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83b666c6_a8c0_482d_8db0_f596b3cf70de.slice/crio-bd7519af53a97078c6e8b67ad5747385027926d11ff6e085bed9d478e143a2a9 WatchSource:0}: Error finding container bd7519af53a97078c6e8b67ad5747385027926d11ff6e085bed9d478e143a2a9: Status 404 returned error can't find the container with id bd7519af53a97078c6e8b67ad5747385027926d11ff6e085bed9d478e143a2a9 Jul 06 19:35:51.659498 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:51.659473 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57b67684d-5s9gj" event={"ID":"83b666c6-a8c0-482d-8db0-f596b3cf70de","Type":"ContainerStarted","Data":"bd7519af53a97078c6e8b67ad5747385027926d11ff6e085bed9d478e143a2a9"} Jul 06 19:35:51.694642 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:51.694591 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-89b4bf89b-dthvn" Jul 06 19:35:52.037707 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:52.037685 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-89b4bf89b-dthvn"] Jul 06 19:35:52.038837 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:35:52.038811 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dc9ce5e_0be5_4f33_8d82_70b915ba1caa.slice/crio-4ac6386ff7d81b66421e672bf0db175eb6d29f3c0e67e818dc6888f7545cbcce WatchSource:0}: Error finding container 4ac6386ff7d81b66421e672bf0db175eb6d29f3c0e67e818dc6888f7545cbcce: Status 404 returned error can't find the container with id 4ac6386ff7d81b66421e672bf0db175eb6d29f3c0e67e818dc6888f7545cbcce Jul 06 19:35:52.670055 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:52.670009 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-89b4bf89b-dthvn" event={"ID":"1dc9ce5e-0be5-4f33-8d82-70b915ba1caa","Type":"ContainerStarted","Data":"4ac6386ff7d81b66421e672bf0db175eb6d29f3c0e67e818dc6888f7545cbcce"} Jul 06 19:35:54.687979 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:54.687942 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57b67684d-5s9gj" event={"ID":"83b666c6-a8c0-482d-8db0-f596b3cf70de","Type":"ContainerStarted","Data":"7db1036769e785f25f4854679ffa808cd3d3f85eb4fef7e78fe3bafb60b4fd4c"} Jul 06 19:35:54.688423 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:54.688010 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-57b67684d-5s9gj" Jul 06 19:35:54.689706 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:54.689674 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-89b4bf89b-dthvn" event={"ID":"1dc9ce5e-0be5-4f33-8d82-70b915ba1caa","Type":"ContainerStarted","Data":"fdeb1a34a2dc6bcd9e6793c2e8826c938f1d2cf80c888a6e0f26c09a83c5983b"} Jul 06 19:35:54.689848 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:54.689731 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-89b4bf89b-dthvn" Jul 06 19:35:54.716577 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:54.716531 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-57b67684d-5s9gj" podStartSLOduration=1.413565078 podStartE2EDuration="3.716517632s" totalImagesPullingTime="2.302952554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:35:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:35:54.713585535 +0000 UTC m=+563.533674247" watchObservedRunningTime="2026-07-06 19:35:54.716517632 +0000 UTC m=+563.536606343" Jul 06 19:35:54.741900 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:35:54.741863 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-89b4bf89b-dthvn" podStartSLOduration=1.8215490170000002 podStartE2EDuration="3.741847737s" totalImagesPullingTime="1.92029872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:35:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:35:54.741326026 +0000 UTC m=+563.561414758" watchObservedRunningTime="2026-07-06 19:35:54.741847737 +0000 UTC m=+563.561936448" Jul 06 19:36:05.703087 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:05.703049 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-57b67684d-5s9gj" Jul 06 19:36:05.703519 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:05.703359 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-89b4bf89b-dthvn" Jul 06 19:36:05.785146 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:05.785115 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57b67684d-5s9gj"] Jul 06 19:36:05.785367 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:05.785332 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-57b67684d-5s9gj" podUID="83b666c6-a8c0-482d-8db0-f596b3cf70de" containerName="manager" containerID="cri-o://7db1036769e785f25f4854679ffa808cd3d3f85eb4fef7e78fe3bafb60b4fd4c" gracePeriod=30 Jul 06 19:36:06.029988 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:06.029966 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57b67684d-5s9gj" Jul 06 19:36:06.063090 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:06.063070 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/83b666c6-a8c0-482d-8db0-f596b3cf70de-webhook-cert\") pod \"83b666c6-a8c0-482d-8db0-f596b3cf70de\" (UID: \"83b666c6-a8c0-482d-8db0-f596b3cf70de\") " Jul 06 19:36:06.063233 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:06.063105 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzqd8\" (UniqueName: \"kubernetes.io/projected/83b666c6-a8c0-482d-8db0-f596b3cf70de-kube-api-access-hzqd8\") pod \"83b666c6-a8c0-482d-8db0-f596b3cf70de\" (UID: \"83b666c6-a8c0-482d-8db0-f596b3cf70de\") " Jul 06 19:36:06.065139 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:06.065107 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83b666c6-a8c0-482d-8db0-f596b3cf70de-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "83b666c6-a8c0-482d-8db0-f596b3cf70de" (UID: "83b666c6-a8c0-482d-8db0-f596b3cf70de"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:36:06.065257 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:06.065158 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83b666c6-a8c0-482d-8db0-f596b3cf70de-kube-api-access-hzqd8" (OuterVolumeSpecName: "kube-api-access-hzqd8") pod "83b666c6-a8c0-482d-8db0-f596b3cf70de" (UID: "83b666c6-a8c0-482d-8db0-f596b3cf70de"). InnerVolumeSpecName "kube-api-access-hzqd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:36:06.164133 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:06.164113 2575 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/83b666c6-a8c0-482d-8db0-f596b3cf70de-webhook-cert\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:36:06.164133 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:06.164134 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzqd8\" (UniqueName: \"kubernetes.io/projected/83b666c6-a8c0-482d-8db0-f596b3cf70de-kube-api-access-hzqd8\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:36:06.763100 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:06.763066 2575 generic.go:356] "Generic (PLEG): container finished" podID="83b666c6-a8c0-482d-8db0-f596b3cf70de" containerID="7db1036769e785f25f4854679ffa808cd3d3f85eb4fef7e78fe3bafb60b4fd4c" exitCode=0 Jul 06 19:36:06.763583 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:06.763103 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57b67684d-5s9gj" Jul 06 19:36:06.763583 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:06.763148 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57b67684d-5s9gj" event={"ID":"83b666c6-a8c0-482d-8db0-f596b3cf70de","Type":"ContainerDied","Data":"7db1036769e785f25f4854679ffa808cd3d3f85eb4fef7e78fe3bafb60b4fd4c"} Jul 06 19:36:06.763583 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:06.763223 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57b67684d-5s9gj" event={"ID":"83b666c6-a8c0-482d-8db0-f596b3cf70de","Type":"ContainerDied","Data":"bd7519af53a97078c6e8b67ad5747385027926d11ff6e085bed9d478e143a2a9"} Jul 06 19:36:06.763583 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:06.763247 2575 scope.go:117] "RemoveContainer" containerID="7db1036769e785f25f4854679ffa808cd3d3f85eb4fef7e78fe3bafb60b4fd4c" Jul 06 19:36:06.773267 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:06.773252 2575 scope.go:117] "RemoveContainer" containerID="7db1036769e785f25f4854679ffa808cd3d3f85eb4fef7e78fe3bafb60b4fd4c" Jul 06 19:36:06.773516 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:36:06.773499 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7db1036769e785f25f4854679ffa808cd3d3f85eb4fef7e78fe3bafb60b4fd4c\": container with ID starting with 7db1036769e785f25f4854679ffa808cd3d3f85eb4fef7e78fe3bafb60b4fd4c not found: ID does not exist" containerID="7db1036769e785f25f4854679ffa808cd3d3f85eb4fef7e78fe3bafb60b4fd4c" Jul 06 19:36:06.773578 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:06.773529 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7db1036769e785f25f4854679ffa808cd3d3f85eb4fef7e78fe3bafb60b4fd4c"} err="failed to get container status \"7db1036769e785f25f4854679ffa808cd3d3f85eb4fef7e78fe3bafb60b4fd4c\": rpc error: code = NotFound desc = could not find container \"7db1036769e785f25f4854679ffa808cd3d3f85eb4fef7e78fe3bafb60b4fd4c\": container with ID starting with 7db1036769e785f25f4854679ffa808cd3d3f85eb4fef7e78fe3bafb60b4fd4c not found: ID does not exist" Jul 06 19:36:06.787934 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:06.787908 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57b67684d-5s9gj"] Jul 06 19:36:06.799881 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:06.799858 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-57b67684d-5s9gj"] Jul 06 19:36:07.794734 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:07.794704 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83b666c6-a8c0-482d-8db0-f596b3cf70de" path="/var/lib/kubelet/pods/83b666c6-a8c0-482d-8db0-f596b3cf70de/volumes" Jul 06 19:36:24.473940 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:24.473906 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-6wbdc"] Jul 06 19:36:24.474426 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:24.474362 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83b666c6-a8c0-482d-8db0-f596b3cf70de" containerName="manager" Jul 06 19:36:24.474426 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:24.474374 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="83b666c6-a8c0-482d-8db0-f596b3cf70de" containerName="manager" Jul 06 19:36:24.474504 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:24.474458 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="83b666c6-a8c0-482d-8db0-f596b3cf70de" containerName="manager" Jul 06 19:36:24.477145 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:24.477130 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-6wbdc" Jul 06 19:36:24.480159 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:24.480134 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 06 19:36:24.480332 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:24.480173 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-dp8sp\"" Jul 06 19:36:24.495767 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:24.495743 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-6wbdc"] Jul 06 19:36:24.613081 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:24.613050 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql4ll\" (UniqueName: \"kubernetes.io/projected/dd268b94-55ca-4307-beb5-29fdc2d18c78-kube-api-access-ql4ll\") pod \"payload-pre-processing-7964457696-6wbdc\" (UID: \"dd268b94-55ca-4307-beb5-29fdc2d18c78\") " pod="openshift-ingress/payload-pre-processing-7964457696-6wbdc" Jul 06 19:36:24.613081 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:24.613086 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd268b94-55ca-4307-beb5-29fdc2d18c78-plugins-config-volume\") pod \"payload-pre-processing-7964457696-6wbdc\" (UID: \"dd268b94-55ca-4307-beb5-29fdc2d18c78\") " pod="openshift-ingress/payload-pre-processing-7964457696-6wbdc" Jul 06 19:36:24.713753 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:24.713715 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql4ll\" (UniqueName: \"kubernetes.io/projected/dd268b94-55ca-4307-beb5-29fdc2d18c78-kube-api-access-ql4ll\") pod \"payload-pre-processing-7964457696-6wbdc\" (UID: \"dd268b94-55ca-4307-beb5-29fdc2d18c78\") " pod="openshift-ingress/payload-pre-processing-7964457696-6wbdc" Jul 06 19:36:24.713944 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:24.713767 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd268b94-55ca-4307-beb5-29fdc2d18c78-plugins-config-volume\") pod \"payload-pre-processing-7964457696-6wbdc\" (UID: \"dd268b94-55ca-4307-beb5-29fdc2d18c78\") " pod="openshift-ingress/payload-pre-processing-7964457696-6wbdc" Jul 06 19:36:24.714484 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:24.714464 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd268b94-55ca-4307-beb5-29fdc2d18c78-plugins-config-volume\") pod \"payload-pre-processing-7964457696-6wbdc\" (UID: \"dd268b94-55ca-4307-beb5-29fdc2d18c78\") " pod="openshift-ingress/payload-pre-processing-7964457696-6wbdc" Jul 06 19:36:24.725595 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:24.725543 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql4ll\" (UniqueName: \"kubernetes.io/projected/dd268b94-55ca-4307-beb5-29fdc2d18c78-kube-api-access-ql4ll\") pod \"payload-pre-processing-7964457696-6wbdc\" (UID: \"dd268b94-55ca-4307-beb5-29fdc2d18c78\") " pod="openshift-ingress/payload-pre-processing-7964457696-6wbdc" Jul 06 19:36:24.788554 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:24.788530 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-6wbdc" Jul 06 19:36:24.926652 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:24.926628 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-6wbdc"] Jul 06 19:36:24.929547 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:36:24.929521 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd268b94_55ca_4307_beb5_29fdc2d18c78.slice/crio-d7c524225978ef4a08cdf512d3ca046b4cbd8619579c76ca868e007c74199367 WatchSource:0}: Error finding container d7c524225978ef4a08cdf512d3ca046b4cbd8619579c76ca868e007c74199367: Status 404 returned error can't find the container with id d7c524225978ef4a08cdf512d3ca046b4cbd8619579c76ca868e007c74199367 Jul 06 19:36:25.890140 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:25.890103 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-6wbdc" event={"ID":"dd268b94-55ca-4307-beb5-29fdc2d18c78","Type":"ContainerStarted","Data":"d7c524225978ef4a08cdf512d3ca046b4cbd8619579c76ca868e007c74199367"} Jul 06 19:36:26.898299 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:26.898267 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-6wbdc" event={"ID":"dd268b94-55ca-4307-beb5-29fdc2d18c78","Type":"ContainerStarted","Data":"a77f210a7c9d5608b1f043df229403ab441371313d7bf52de0b5b9ed453764f5"} Jul 06 19:36:26.898705 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:26.898494 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-6wbdc" Jul 06 19:36:26.920106 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:26.920067 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-6wbdc" podStartSLOduration=1.7209837000000001 podStartE2EDuration="2.920054384s" totalImagesPullingTime="1.199070684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:36:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:36:26.917162079 +0000 UTC m=+595.737250789" watchObservedRunningTime="2026-07-06 19:36:26.920054384 +0000 UTC m=+595.740143094" Jul 06 19:36:37.906322 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:37.906291 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-6wbdc" Jul 06 19:36:48.646758 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:48.646727 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c"] Jul 06 19:36:48.651324 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:48.651306 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c" Jul 06 19:36:48.655288 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:48.655266 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 06 19:36:48.655414 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:48.655298 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 06 19:36:48.655414 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:48.655276 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 06 19:36:48.655414 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:48.655269 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-cqnv7\"" Jul 06 19:36:48.661447 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:48.661427 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c"] Jul 06 19:36:48.819107 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:48.819077 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjgkr\" (UniqueName: \"kubernetes.io/projected/4355afe1-da7d-4bee-9033-1730e8b75251-kube-api-access-kjgkr\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c\" (UID: \"4355afe1-da7d-4bee-9033-1730e8b75251\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c" Jul 06 19:36:48.819287 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:48.819111 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4355afe1-da7d-4bee-9033-1730e8b75251-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c\" (UID: \"4355afe1-da7d-4bee-9033-1730e8b75251\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c" Jul 06 19:36:48.819287 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:48.819142 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4355afe1-da7d-4bee-9033-1730e8b75251-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c\" (UID: \"4355afe1-da7d-4bee-9033-1730e8b75251\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c" Jul 06 19:36:48.819365 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:48.819294 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4355afe1-da7d-4bee-9033-1730e8b75251-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c\" (UID: \"4355afe1-da7d-4bee-9033-1730e8b75251\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c" Jul 06 19:36:48.819365 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:48.819313 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4355afe1-da7d-4bee-9033-1730e8b75251-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c\" (UID: \"4355afe1-da7d-4bee-9033-1730e8b75251\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c" Jul 06 19:36:48.819365 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:48.819343 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4355afe1-da7d-4bee-9033-1730e8b75251-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c\" (UID: \"4355afe1-da7d-4bee-9033-1730e8b75251\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c" Jul 06 19:36:48.920474 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:48.920386 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4355afe1-da7d-4bee-9033-1730e8b75251-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c\" (UID: \"4355afe1-da7d-4bee-9033-1730e8b75251\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c" Jul 06 19:36:48.920474 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:48.920425 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4355afe1-da7d-4bee-9033-1730e8b75251-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c\" (UID: \"4355afe1-da7d-4bee-9033-1730e8b75251\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c" Jul 06 19:36:48.920474 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:48.920470 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4355afe1-da7d-4bee-9033-1730e8b75251-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c\" (UID: \"4355afe1-da7d-4bee-9033-1730e8b75251\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c" Jul 06 19:36:48.920734 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:48.920518 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjgkr\" (UniqueName: \"kubernetes.io/projected/4355afe1-da7d-4bee-9033-1730e8b75251-kube-api-access-kjgkr\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c\" (UID: \"4355afe1-da7d-4bee-9033-1730e8b75251\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c" Jul 06 19:36:48.920734 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:48.920547 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4355afe1-da7d-4bee-9033-1730e8b75251-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c\" (UID: \"4355afe1-da7d-4bee-9033-1730e8b75251\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c" Jul 06 19:36:48.920734 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:48.920591 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4355afe1-da7d-4bee-9033-1730e8b75251-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c\" (UID: \"4355afe1-da7d-4bee-9033-1730e8b75251\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c" Jul 06 19:36:48.920878 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:48.920834 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4355afe1-da7d-4bee-9033-1730e8b75251-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c\" (UID: \"4355afe1-da7d-4bee-9033-1730e8b75251\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c" Jul 06 19:36:48.920927 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:48.920887 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4355afe1-da7d-4bee-9033-1730e8b75251-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c\" (UID: \"4355afe1-da7d-4bee-9033-1730e8b75251\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c" Jul 06 19:36:48.920927 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:48.920891 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4355afe1-da7d-4bee-9033-1730e8b75251-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c\" (UID: \"4355afe1-da7d-4bee-9033-1730e8b75251\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c" Jul 06 19:36:48.922839 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:48.922809 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4355afe1-da7d-4bee-9033-1730e8b75251-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c\" (UID: \"4355afe1-da7d-4bee-9033-1730e8b75251\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c" Jul 06 19:36:48.923100 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:48.923082 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4355afe1-da7d-4bee-9033-1730e8b75251-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c\" (UID: \"4355afe1-da7d-4bee-9033-1730e8b75251\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c" Jul 06 19:36:48.937103 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:48.937084 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjgkr\" (UniqueName: \"kubernetes.io/projected/4355afe1-da7d-4bee-9033-1730e8b75251-kube-api-access-kjgkr\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c\" (UID: \"4355afe1-da7d-4bee-9033-1730e8b75251\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c" Jul 06 19:36:48.961939 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:48.961918 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c" Jul 06 19:36:49.112555 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:49.112526 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c"] Jul 06 19:36:49.113691 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:36:49.113666 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4355afe1_da7d_4bee_9033_1730e8b75251.slice/crio-14e474fd82b0fc818f6875ccfd4dd78b9a0091599831c9c8e1fe8e93cb83a744 WatchSource:0}: Error finding container 14e474fd82b0fc818f6875ccfd4dd78b9a0091599831c9c8e1fe8e93cb83a744: Status 404 returned error can't find the container with id 14e474fd82b0fc818f6875ccfd4dd78b9a0091599831c9c8e1fe8e93cb83a744 Jul 06 19:36:49.114985 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:49.114967 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:36:50.037599 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:50.037553 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c" event={"ID":"4355afe1-da7d-4bee-9033-1730e8b75251","Type":"ContainerStarted","Data":"14e474fd82b0fc818f6875ccfd4dd78b9a0091599831c9c8e1fe8e93cb83a744"} Jul 06 19:36:51.044577 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:51.044545 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c" event={"ID":"4355afe1-da7d-4bee-9033-1730e8b75251","Type":"ContainerStarted","Data":"880546b4a77bb196057851c898acf8fd8659fc11b4fe3c2419c9fc675cf322b4"} Jul 06 19:36:51.068790 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:51.068736 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c" podStartSLOduration=1.928130854 podStartE2EDuration="3.068716747s" totalImagesPullingTime="1.140585893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:36:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:36:51.065600645 +0000 UTC m=+619.885689372" watchObservedRunningTime="2026-07-06 19:36:51.068716747 +0000 UTC m=+619.888805464" Jul 06 19:36:54.837413 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:54.837339 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-zjfn7"] Jul 06 19:36:54.841411 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:54.841388 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjfn7" Jul 06 19:36:54.844017 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:54.843994 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 06 19:36:54.851304 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:54.851282 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-zjfn7"] Jul 06 19:36:54.864692 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:54.864654 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zk8b\" (UniqueName: \"kubernetes.io/projected/4ca952cf-2db3-4a5a-9b7f-b1308373192e-kube-api-access-9zk8b\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zjfn7\" (UID: \"4ca952cf-2db3-4a5a-9b7f-b1308373192e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjfn7" Jul 06 19:36:54.864809 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:54.864701 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ca952cf-2db3-4a5a-9b7f-b1308373192e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zjfn7\" (UID: \"4ca952cf-2db3-4a5a-9b7f-b1308373192e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjfn7" Jul 06 19:36:54.864809 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:54.864797 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ca952cf-2db3-4a5a-9b7f-b1308373192e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zjfn7\" (UID: \"4ca952cf-2db3-4a5a-9b7f-b1308373192e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjfn7" Jul 06 19:36:54.864931 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:54.864842 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ca952cf-2db3-4a5a-9b7f-b1308373192e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zjfn7\" (UID: \"4ca952cf-2db3-4a5a-9b7f-b1308373192e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjfn7" Jul 06 19:36:54.864991 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:54.864921 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ca952cf-2db3-4a5a-9b7f-b1308373192e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zjfn7\" (UID: \"4ca952cf-2db3-4a5a-9b7f-b1308373192e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjfn7" Jul 06 19:36:54.864991 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:54.864962 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ca952cf-2db3-4a5a-9b7f-b1308373192e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zjfn7\" (UID: \"4ca952cf-2db3-4a5a-9b7f-b1308373192e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjfn7" Jul 06 19:36:54.965999 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:54.965969 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ca952cf-2db3-4a5a-9b7f-b1308373192e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zjfn7\" (UID: \"4ca952cf-2db3-4a5a-9b7f-b1308373192e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjfn7" Jul 06 19:36:54.966125 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:54.966003 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ca952cf-2db3-4a5a-9b7f-b1308373192e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zjfn7\" (UID: \"4ca952cf-2db3-4a5a-9b7f-b1308373192e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjfn7" Jul 06 19:36:54.966125 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:54.966022 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ca952cf-2db3-4a5a-9b7f-b1308373192e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zjfn7\" (UID: \"4ca952cf-2db3-4a5a-9b7f-b1308373192e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjfn7" Jul 06 19:36:54.966125 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:54.966058 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zk8b\" (UniqueName: \"kubernetes.io/projected/4ca952cf-2db3-4a5a-9b7f-b1308373192e-kube-api-access-9zk8b\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zjfn7\" (UID: \"4ca952cf-2db3-4a5a-9b7f-b1308373192e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjfn7" Jul 06 19:36:54.966125 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:54.966087 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ca952cf-2db3-4a5a-9b7f-b1308373192e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zjfn7\" (UID: \"4ca952cf-2db3-4a5a-9b7f-b1308373192e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjfn7" Jul 06 19:36:54.966398 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:54.966142 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ca952cf-2db3-4a5a-9b7f-b1308373192e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zjfn7\" (UID: \"4ca952cf-2db3-4a5a-9b7f-b1308373192e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjfn7" Jul 06 19:36:54.966398 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:54.966389 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ca952cf-2db3-4a5a-9b7f-b1308373192e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zjfn7\" (UID: \"4ca952cf-2db3-4a5a-9b7f-b1308373192e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjfn7" Jul 06 19:36:54.966626 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:54.966467 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ca952cf-2db3-4a5a-9b7f-b1308373192e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zjfn7\" (UID: \"4ca952cf-2db3-4a5a-9b7f-b1308373192e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjfn7" Jul 06 19:36:54.966626 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:54.966487 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ca952cf-2db3-4a5a-9b7f-b1308373192e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zjfn7\" (UID: \"4ca952cf-2db3-4a5a-9b7f-b1308373192e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjfn7" Jul 06 19:36:54.968550 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:54.968525 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ca952cf-2db3-4a5a-9b7f-b1308373192e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zjfn7\" (UID: \"4ca952cf-2db3-4a5a-9b7f-b1308373192e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjfn7" Jul 06 19:36:54.968808 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:54.968790 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ca952cf-2db3-4a5a-9b7f-b1308373192e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zjfn7\" (UID: \"4ca952cf-2db3-4a5a-9b7f-b1308373192e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjfn7" Jul 06 19:36:54.978818 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:54.978796 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zk8b\" (UniqueName: \"kubernetes.io/projected/4ca952cf-2db3-4a5a-9b7f-b1308373192e-kube-api-access-9zk8b\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zjfn7\" (UID: \"4ca952cf-2db3-4a5a-9b7f-b1308373192e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjfn7" Jul 06 19:36:55.151863 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:55.151807 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjfn7" Jul 06 19:36:55.294800 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:55.294764 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-zjfn7"] Jul 06 19:36:55.298744 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:36:55.298716 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca952cf_2db3_4a5a_9b7f_b1308373192e.slice/crio-9e604bfde6080480ffcc94c8deb276330ab7ee9ac5cb619c5e6f23e21c8b12d7 WatchSource:0}: Error finding container 9e604bfde6080480ffcc94c8deb276330ab7ee9ac5cb619c5e6f23e21c8b12d7: Status 404 returned error can't find the container with id 9e604bfde6080480ffcc94c8deb276330ab7ee9ac5cb619c5e6f23e21c8b12d7 Jul 06 19:36:56.077157 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:56.077117 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjfn7" event={"ID":"4ca952cf-2db3-4a5a-9b7f-b1308373192e","Type":"ContainerStarted","Data":"98a12595e2c7b0e6ea3d007427a7ae35f85b157258cfecbcd8711e4625391fed"} Jul 06 19:36:56.077716 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:56.077202 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjfn7" event={"ID":"4ca952cf-2db3-4a5a-9b7f-b1308373192e","Type":"ContainerStarted","Data":"9e604bfde6080480ffcc94c8deb276330ab7ee9ac5cb619c5e6f23e21c8b12d7"} Jul 06 19:36:56.106157 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:56.106098 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjfn7" podStartSLOduration=1.887485738 podStartE2EDuration="2.106078393s" totalImagesPullingTime="218.592655ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:36:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:36:56.102378063 +0000 UTC m=+624.922466776" watchObservedRunningTime="2026-07-06 19:36:56.106078393 +0000 UTC m=+624.926167105" Jul 06 19:36:58.962273 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:58.962242 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c" Jul 06 19:36:58.962654 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:58.962292 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c" Jul 06 19:36:58.974584 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:58.974561 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c" Jul 06 19:36:59.104883 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:36:59.104861 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c" Jul 06 19:37:05.152669 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:37:05.152639 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjfn7" Jul 06 19:37:05.153032 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:37:05.152706 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjfn7" Jul 06 19:37:05.164986 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:37:05.164966 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjfn7" Jul 06 19:37:06.150246 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:37:06.150218 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zjfn7" Jul 06 19:38:29.454879 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:38:29.454796 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6ffb7848c6-qjd89"] Jul 06 19:38:29.455375 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:38:29.455025 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-6ffb7848c6-qjd89" podUID="67a17400-c96e-4ed6-9c7a-bfe7c5a8ee49" containerName="authorino" containerID="cri-o://5a198627b47e6fdf90d34ea59991255d3453a6ba8004b5657bbc6fd547088ba3" gracePeriod=30 Jul 06 19:38:29.666224 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:38:29.666175 2575 generic.go:356] "Generic (PLEG): container finished" podID="67a17400-c96e-4ed6-9c7a-bfe7c5a8ee49" containerID="5a198627b47e6fdf90d34ea59991255d3453a6ba8004b5657bbc6fd547088ba3" exitCode=0 Jul 06 19:38:29.666387 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:38:29.666226 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6ffb7848c6-qjd89" event={"ID":"67a17400-c96e-4ed6-9c7a-bfe7c5a8ee49","Type":"ContainerDied","Data":"5a198627b47e6fdf90d34ea59991255d3453a6ba8004b5657bbc6fd547088ba3"} Jul 06 19:38:29.721570 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:38:29.721549 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6ffb7848c6-qjd89" Jul 06 19:38:29.848770 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:38:29.848741 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shntg\" (UniqueName: \"kubernetes.io/projected/67a17400-c96e-4ed6-9c7a-bfe7c5a8ee49-kube-api-access-shntg\") pod \"67a17400-c96e-4ed6-9c7a-bfe7c5a8ee49\" (UID: \"67a17400-c96e-4ed6-9c7a-bfe7c5a8ee49\") " Jul 06 19:38:29.848925 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:38:29.848783 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/67a17400-c96e-4ed6-9c7a-bfe7c5a8ee49-tls-cert\") pod \"67a17400-c96e-4ed6-9c7a-bfe7c5a8ee49\" (UID: \"67a17400-c96e-4ed6-9c7a-bfe7c5a8ee49\") " Jul 06 19:38:29.850889 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:38:29.850859 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67a17400-c96e-4ed6-9c7a-bfe7c5a8ee49-kube-api-access-shntg" (OuterVolumeSpecName: "kube-api-access-shntg") pod "67a17400-c96e-4ed6-9c7a-bfe7c5a8ee49" (UID: "67a17400-c96e-4ed6-9c7a-bfe7c5a8ee49"). InnerVolumeSpecName "kube-api-access-shntg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:38:29.861944 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:38:29.861925 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67a17400-c96e-4ed6-9c7a-bfe7c5a8ee49-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "67a17400-c96e-4ed6-9c7a-bfe7c5a8ee49" (UID: "67a17400-c96e-4ed6-9c7a-bfe7c5a8ee49"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:38:29.950200 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:38:29.950158 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shntg\" (UniqueName: \"kubernetes.io/projected/67a17400-c96e-4ed6-9c7a-bfe7c5a8ee49-kube-api-access-shntg\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:38:29.950300 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:38:29.950205 2575 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/67a17400-c96e-4ed6-9c7a-bfe7c5a8ee49-tls-cert\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:38:30.673120 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:38:30.673094 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6ffb7848c6-qjd89" Jul 06 19:38:30.673522 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:38:30.673090 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6ffb7848c6-qjd89" event={"ID":"67a17400-c96e-4ed6-9c7a-bfe7c5a8ee49","Type":"ContainerDied","Data":"51c7b015d7e7fadc5e8e2438ed80fc60c7e4c7ec66b930878e6700ca75008389"} Jul 06 19:38:30.673522 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:38:30.673245 2575 scope.go:117] "RemoveContainer" containerID="5a198627b47e6fdf90d34ea59991255d3453a6ba8004b5657bbc6fd547088ba3" Jul 06 19:38:30.696720 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:38:30.696691 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6ffb7848c6-qjd89"] Jul 06 19:38:30.701937 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:38:30.701916 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6ffb7848c6-qjd89"] Jul 06 19:38:31.794929 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:38:31.794900 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67a17400-c96e-4ed6-9c7a-bfe7c5a8ee49" path="/var/lib/kubelet/pods/67a17400-c96e-4ed6-9c7a-bfe7c5a8ee49/volumes" Jul 06 19:39:26.527892 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:26.527862 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c9cb4cdbf-zcnzl"] Jul 06 19:39:26.528430 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:26.528415 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67a17400-c96e-4ed6-9c7a-bfe7c5a8ee49" containerName="authorino" Jul 06 19:39:26.528430 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:26.528432 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="67a17400-c96e-4ed6-9c7a-bfe7c5a8ee49" containerName="authorino" Jul 06 19:39:26.528511 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:26.528498 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="67a17400-c96e-4ed6-9c7a-bfe7c5a8ee49" containerName="authorino" Jul 06 19:39:26.531534 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:26.531520 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c9cb4cdbf-zcnzl" Jul 06 19:39:26.556762 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:26.556734 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c9cb4cdbf-zcnzl"] Jul 06 19:39:26.598475 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:26.598451 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6f298902-626e-4071-ab20-1629b3b97359-console-oauth-config\") pod \"console-7c9cb4cdbf-zcnzl\" (UID: \"6f298902-626e-4071-ab20-1629b3b97359\") " pod="openshift-console/console-7c9cb4cdbf-zcnzl" Jul 06 19:39:26.598584 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:26.598493 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6f298902-626e-4071-ab20-1629b3b97359-oauth-serving-cert\") pod \"console-7c9cb4cdbf-zcnzl\" (UID: \"6f298902-626e-4071-ab20-1629b3b97359\") " pod="openshift-console/console-7c9cb4cdbf-zcnzl" Jul 06 19:39:26.598584 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:26.598517 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrg8z\" (UniqueName: \"kubernetes.io/projected/6f298902-626e-4071-ab20-1629b3b97359-kube-api-access-wrg8z\") pod \"console-7c9cb4cdbf-zcnzl\" (UID: \"6f298902-626e-4071-ab20-1629b3b97359\") " pod="openshift-console/console-7c9cb4cdbf-zcnzl" Jul 06 19:39:26.598659 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:26.598592 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6f298902-626e-4071-ab20-1629b3b97359-console-config\") pod \"console-7c9cb4cdbf-zcnzl\" (UID: \"6f298902-626e-4071-ab20-1629b3b97359\") " pod="openshift-console/console-7c9cb4cdbf-zcnzl" Jul 06 19:39:26.598659 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:26.598627 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f298902-626e-4071-ab20-1629b3b97359-trusted-ca-bundle\") pod \"console-7c9cb4cdbf-zcnzl\" (UID: \"6f298902-626e-4071-ab20-1629b3b97359\") " pod="openshift-console/console-7c9cb4cdbf-zcnzl" Jul 06 19:39:26.598659 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:26.598648 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6f298902-626e-4071-ab20-1629b3b97359-console-serving-cert\") pod \"console-7c9cb4cdbf-zcnzl\" (UID: \"6f298902-626e-4071-ab20-1629b3b97359\") " pod="openshift-console/console-7c9cb4cdbf-zcnzl" Jul 06 19:39:26.598795 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:26.598716 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6f298902-626e-4071-ab20-1629b3b97359-service-ca\") pod \"console-7c9cb4cdbf-zcnzl\" (UID: \"6f298902-626e-4071-ab20-1629b3b97359\") " pod="openshift-console/console-7c9cb4cdbf-zcnzl" Jul 06 19:39:26.699425 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:26.699403 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6f298902-626e-4071-ab20-1629b3b97359-console-oauth-config\") pod \"console-7c9cb4cdbf-zcnzl\" (UID: \"6f298902-626e-4071-ab20-1629b3b97359\") " pod="openshift-console/console-7c9cb4cdbf-zcnzl" Jul 06 19:39:26.699559 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:26.699439 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6f298902-626e-4071-ab20-1629b3b97359-oauth-serving-cert\") pod \"console-7c9cb4cdbf-zcnzl\" (UID: \"6f298902-626e-4071-ab20-1629b3b97359\") " pod="openshift-console/console-7c9cb4cdbf-zcnzl" Jul 06 19:39:26.699559 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:26.699460 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrg8z\" (UniqueName: \"kubernetes.io/projected/6f298902-626e-4071-ab20-1629b3b97359-kube-api-access-wrg8z\") pod \"console-7c9cb4cdbf-zcnzl\" (UID: \"6f298902-626e-4071-ab20-1629b3b97359\") " pod="openshift-console/console-7c9cb4cdbf-zcnzl" Jul 06 19:39:26.699559 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:26.699492 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6f298902-626e-4071-ab20-1629b3b97359-console-config\") pod \"console-7c9cb4cdbf-zcnzl\" (UID: \"6f298902-626e-4071-ab20-1629b3b97359\") " pod="openshift-console/console-7c9cb4cdbf-zcnzl" Jul 06 19:39:26.699727 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:26.699601 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f298902-626e-4071-ab20-1629b3b97359-trusted-ca-bundle\") pod \"console-7c9cb4cdbf-zcnzl\" (UID: \"6f298902-626e-4071-ab20-1629b3b97359\") " pod="openshift-console/console-7c9cb4cdbf-zcnzl" Jul 06 19:39:26.699727 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:26.699639 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6f298902-626e-4071-ab20-1629b3b97359-console-serving-cert\") pod \"console-7c9cb4cdbf-zcnzl\" (UID: \"6f298902-626e-4071-ab20-1629b3b97359\") " pod="openshift-console/console-7c9cb4cdbf-zcnzl" Jul 06 19:39:26.699727 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:26.699686 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6f298902-626e-4071-ab20-1629b3b97359-service-ca\") pod \"console-7c9cb4cdbf-zcnzl\" (UID: \"6f298902-626e-4071-ab20-1629b3b97359\") " pod="openshift-console/console-7c9cb4cdbf-zcnzl" Jul 06 19:39:26.700208 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:26.700160 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6f298902-626e-4071-ab20-1629b3b97359-console-config\") pod \"console-7c9cb4cdbf-zcnzl\" (UID: \"6f298902-626e-4071-ab20-1629b3b97359\") " pod="openshift-console/console-7c9cb4cdbf-zcnzl" Jul 06 19:39:26.700310 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:26.700259 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6f298902-626e-4071-ab20-1629b3b97359-oauth-serving-cert\") pod \"console-7c9cb4cdbf-zcnzl\" (UID: \"6f298902-626e-4071-ab20-1629b3b97359\") " pod="openshift-console/console-7c9cb4cdbf-zcnzl" Jul 06 19:39:26.700368 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:26.700317 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6f298902-626e-4071-ab20-1629b3b97359-service-ca\") pod \"console-7c9cb4cdbf-zcnzl\" (UID: \"6f298902-626e-4071-ab20-1629b3b97359\") " pod="openshift-console/console-7c9cb4cdbf-zcnzl" Jul 06 19:39:26.700463 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:26.700441 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f298902-626e-4071-ab20-1629b3b97359-trusted-ca-bundle\") pod \"console-7c9cb4cdbf-zcnzl\" (UID: \"6f298902-626e-4071-ab20-1629b3b97359\") " pod="openshift-console/console-7c9cb4cdbf-zcnzl" Jul 06 19:39:26.701868 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:26.701849 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6f298902-626e-4071-ab20-1629b3b97359-console-serving-cert\") pod \"console-7c9cb4cdbf-zcnzl\" (UID: \"6f298902-626e-4071-ab20-1629b3b97359\") " pod="openshift-console/console-7c9cb4cdbf-zcnzl" Jul 06 19:39:26.702000 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:26.701981 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6f298902-626e-4071-ab20-1629b3b97359-console-oauth-config\") pod \"console-7c9cb4cdbf-zcnzl\" (UID: \"6f298902-626e-4071-ab20-1629b3b97359\") " pod="openshift-console/console-7c9cb4cdbf-zcnzl" Jul 06 19:39:26.711160 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:26.711134 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrg8z\" (UniqueName: \"kubernetes.io/projected/6f298902-626e-4071-ab20-1629b3b97359-kube-api-access-wrg8z\") pod \"console-7c9cb4cdbf-zcnzl\" (UID: \"6f298902-626e-4071-ab20-1629b3b97359\") " pod="openshift-console/console-7c9cb4cdbf-zcnzl" Jul 06 19:39:26.839994 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:26.839967 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c9cb4cdbf-zcnzl" Jul 06 19:39:26.991351 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:26.991326 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c9cb4cdbf-zcnzl"] Jul 06 19:39:26.992312 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:39:26.992287 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f298902_626e_4071_ab20_1629b3b97359.slice/crio-b4908014450865fc9840026b6501b398a5eb14ed4b486b604e734495c3400ba3 WatchSource:0}: Error finding container b4908014450865fc9840026b6501b398a5eb14ed4b486b604e734495c3400ba3: Status 404 returned error can't find the container with id b4908014450865fc9840026b6501b398a5eb14ed4b486b604e734495c3400ba3 Jul 06 19:39:27.019288 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:27.019257 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c9cb4cdbf-zcnzl" event={"ID":"6f298902-626e-4071-ab20-1629b3b97359","Type":"ContainerStarted","Data":"b4908014450865fc9840026b6501b398a5eb14ed4b486b604e734495c3400ba3"} Jul 06 19:39:28.026776 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:28.026744 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c9cb4cdbf-zcnzl" event={"ID":"6f298902-626e-4071-ab20-1629b3b97359","Type":"ContainerStarted","Data":"f2658cdb281a45a8644feee517690b9a7e8975dc169de17b7aba92ae69363b2c"} Jul 06 19:39:28.049655 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:28.049613 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7c9cb4cdbf-zcnzl" podStartSLOduration=2.049596315 podStartE2EDuration="2.049596315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:39:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:39:28.048236915 +0000 UTC m=+776.868325626" watchObservedRunningTime="2026-07-06 19:39:28.049596315 +0000 UTC m=+776.869685026" Jul 06 19:39:36.840682 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:36.840646 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c9cb4cdbf-zcnzl" Jul 06 19:39:36.841048 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:36.840719 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c9cb4cdbf-zcnzl" Jul 06 19:39:36.845052 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:36.845030 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c9cb4cdbf-zcnzl" Jul 06 19:39:37.086461 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:37.086430 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c9cb4cdbf-zcnzl" Jul 06 19:39:37.150513 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:37.150440 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69ff557b65-zcl5k"] Jul 06 19:39:41.121706 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:41.121677 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-89b4bf89b-dthvn"] Jul 06 19:39:41.122262 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:41.121859 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-89b4bf89b-dthvn" podUID="1dc9ce5e-0be5-4f33-8d82-70b915ba1caa" containerName="manager" containerID="cri-o://fdeb1a34a2dc6bcd9e6793c2e8826c938f1d2cf80c888a6e0f26c09a83c5983b" gracePeriod=30 Jul 06 19:39:41.377333 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:41.377273 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-89b4bf89b-dthvn" Jul 06 19:39:41.527227 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:41.527176 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1dc9ce5e-0be5-4f33-8d82-70b915ba1caa-webhook-cert\") pod \"1dc9ce5e-0be5-4f33-8d82-70b915ba1caa\" (UID: \"1dc9ce5e-0be5-4f33-8d82-70b915ba1caa\") " Jul 06 19:39:41.527398 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:41.527259 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pn8zh\" (UniqueName: \"kubernetes.io/projected/1dc9ce5e-0be5-4f33-8d82-70b915ba1caa-kube-api-access-pn8zh\") pod \"1dc9ce5e-0be5-4f33-8d82-70b915ba1caa\" (UID: \"1dc9ce5e-0be5-4f33-8d82-70b915ba1caa\") " Jul 06 19:39:41.529145 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:41.529121 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dc9ce5e-0be5-4f33-8d82-70b915ba1caa-kube-api-access-pn8zh" (OuterVolumeSpecName: "kube-api-access-pn8zh") pod "1dc9ce5e-0be5-4f33-8d82-70b915ba1caa" (UID: "1dc9ce5e-0be5-4f33-8d82-70b915ba1caa"). InnerVolumeSpecName "kube-api-access-pn8zh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:39:41.529271 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:41.529143 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dc9ce5e-0be5-4f33-8d82-70b915ba1caa-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "1dc9ce5e-0be5-4f33-8d82-70b915ba1caa" (UID: "1dc9ce5e-0be5-4f33-8d82-70b915ba1caa"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:39:41.628575 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:41.628523 2575 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1dc9ce5e-0be5-4f33-8d82-70b915ba1caa-webhook-cert\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:39:41.628575 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:41.628543 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn8zh\" (UniqueName: \"kubernetes.io/projected/1dc9ce5e-0be5-4f33-8d82-70b915ba1caa-kube-api-access-pn8zh\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:39:42.112301 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:42.112270 2575 generic.go:356] "Generic (PLEG): container finished" podID="1dc9ce5e-0be5-4f33-8d82-70b915ba1caa" containerID="fdeb1a34a2dc6bcd9e6793c2e8826c938f1d2cf80c888a6e0f26c09a83c5983b" exitCode=0 Jul 06 19:39:42.112432 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:42.112313 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-89b4bf89b-dthvn" Jul 06 19:39:42.112432 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:42.112351 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-89b4bf89b-dthvn" event={"ID":"1dc9ce5e-0be5-4f33-8d82-70b915ba1caa","Type":"ContainerDied","Data":"fdeb1a34a2dc6bcd9e6793c2e8826c938f1d2cf80c888a6e0f26c09a83c5983b"} Jul 06 19:39:42.112432 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:42.112402 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-89b4bf89b-dthvn" event={"ID":"1dc9ce5e-0be5-4f33-8d82-70b915ba1caa","Type":"ContainerDied","Data":"4ac6386ff7d81b66421e672bf0db175eb6d29f3c0e67e818dc6888f7545cbcce"} Jul 06 19:39:42.112432 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:42.112429 2575 scope.go:117] "RemoveContainer" containerID="fdeb1a34a2dc6bcd9e6793c2e8826c938f1d2cf80c888a6e0f26c09a83c5983b" Jul 06 19:39:42.122235 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:42.122212 2575 scope.go:117] "RemoveContainer" containerID="fdeb1a34a2dc6bcd9e6793c2e8826c938f1d2cf80c888a6e0f26c09a83c5983b" Jul 06 19:39:42.122536 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:39:42.122478 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdeb1a34a2dc6bcd9e6793c2e8826c938f1d2cf80c888a6e0f26c09a83c5983b\": container with ID starting with fdeb1a34a2dc6bcd9e6793c2e8826c938f1d2cf80c888a6e0f26c09a83c5983b not found: ID does not exist" containerID="fdeb1a34a2dc6bcd9e6793c2e8826c938f1d2cf80c888a6e0f26c09a83c5983b" Jul 06 19:39:42.122536 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:42.122502 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdeb1a34a2dc6bcd9e6793c2e8826c938f1d2cf80c888a6e0f26c09a83c5983b"} err="failed to get container status \"fdeb1a34a2dc6bcd9e6793c2e8826c938f1d2cf80c888a6e0f26c09a83c5983b\": rpc error: code = NotFound desc = could not find container \"fdeb1a34a2dc6bcd9e6793c2e8826c938f1d2cf80c888a6e0f26c09a83c5983b\": container with ID starting with fdeb1a34a2dc6bcd9e6793c2e8826c938f1d2cf80c888a6e0f26c09a83c5983b not found: ID does not exist" Jul 06 19:39:42.142495 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:42.142468 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-89b4bf89b-dthvn"] Jul 06 19:39:42.149115 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:42.149093 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-89b4bf89b-dthvn"] Jul 06 19:39:42.996980 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:42.996952 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-89b4bf89b-sxltm"] Jul 06 19:39:42.997469 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:42.997450 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dc9ce5e-0be5-4f33-8d82-70b915ba1caa" containerName="manager" Jul 06 19:39:42.997469 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:42.997469 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dc9ce5e-0be5-4f33-8d82-70b915ba1caa" containerName="manager" Jul 06 19:39:42.997621 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:42.997551 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dc9ce5e-0be5-4f33-8d82-70b915ba1caa" containerName="manager" Jul 06 19:39:43.002312 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:43.002292 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-89b4bf89b-sxltm" Jul 06 19:39:43.006240 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:43.006225 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 06 19:39:43.006240 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:43.006234 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nwfhq\"" Jul 06 19:39:43.006365 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:43.006226 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 06 19:39:43.014000 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:43.013977 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-89b4bf89b-sxltm"] Jul 06 19:39:43.140415 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:43.140395 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5b2875d4-ab7f-4bf1-851d-a4006df17049-webhook-cert\") pod \"maas-controller-89b4bf89b-sxltm\" (UID: \"5b2875d4-ab7f-4bf1-851d-a4006df17049\") " pod="opendatahub/maas-controller-89b4bf89b-sxltm" Jul 06 19:39:43.140719 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:43.140462 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mjlp\" (UniqueName: \"kubernetes.io/projected/5b2875d4-ab7f-4bf1-851d-a4006df17049-kube-api-access-4mjlp\") pod \"maas-controller-89b4bf89b-sxltm\" (UID: \"5b2875d4-ab7f-4bf1-851d-a4006df17049\") " pod="opendatahub/maas-controller-89b4bf89b-sxltm" Jul 06 19:39:43.241270 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:43.241244 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mjlp\" (UniqueName: \"kubernetes.io/projected/5b2875d4-ab7f-4bf1-851d-a4006df17049-kube-api-access-4mjlp\") pod \"maas-controller-89b4bf89b-sxltm\" (UID: \"5b2875d4-ab7f-4bf1-851d-a4006df17049\") " pod="opendatahub/maas-controller-89b4bf89b-sxltm" Jul 06 19:39:43.241383 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:43.241287 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5b2875d4-ab7f-4bf1-851d-a4006df17049-webhook-cert\") pod \"maas-controller-89b4bf89b-sxltm\" (UID: \"5b2875d4-ab7f-4bf1-851d-a4006df17049\") " pod="opendatahub/maas-controller-89b4bf89b-sxltm" Jul 06 19:39:43.243551 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:43.243526 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5b2875d4-ab7f-4bf1-851d-a4006df17049-webhook-cert\") pod \"maas-controller-89b4bf89b-sxltm\" (UID: \"5b2875d4-ab7f-4bf1-851d-a4006df17049\") " pod="opendatahub/maas-controller-89b4bf89b-sxltm" Jul 06 19:39:43.252698 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:43.252649 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mjlp\" (UniqueName: \"kubernetes.io/projected/5b2875d4-ab7f-4bf1-851d-a4006df17049-kube-api-access-4mjlp\") pod \"maas-controller-89b4bf89b-sxltm\" (UID: \"5b2875d4-ab7f-4bf1-851d-a4006df17049\") " pod="opendatahub/maas-controller-89b4bf89b-sxltm" Jul 06 19:39:43.312349 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:43.312329 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-89b4bf89b-sxltm" Jul 06 19:39:43.455325 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:43.455290 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-89b4bf89b-sxltm"] Jul 06 19:39:43.456929 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:39:43.456907 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b2875d4_ab7f_4bf1_851d_a4006df17049.slice/crio-7b9c31b501b845c1fe7b49cc83f0aae66415bdf7741b0e64b940ea5e7ba7d882 WatchSource:0}: Error finding container 7b9c31b501b845c1fe7b49cc83f0aae66415bdf7741b0e64b940ea5e7ba7d882: Status 404 returned error can't find the container with id 7b9c31b501b845c1fe7b49cc83f0aae66415bdf7741b0e64b940ea5e7ba7d882 Jul 06 19:39:43.794814 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:43.794787 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dc9ce5e-0be5-4f33-8d82-70b915ba1caa" path="/var/lib/kubelet/pods/1dc9ce5e-0be5-4f33-8d82-70b915ba1caa/volumes" Jul 06 19:39:44.126548 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:44.126519 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-89b4bf89b-sxltm" event={"ID":"5b2875d4-ab7f-4bf1-851d-a4006df17049","Type":"ContainerStarted","Data":"37125d68f3196d24d852c7114858b7634435c3b85f41caf4acd0e13fcd2e2b32"} Jul 06 19:39:44.126665 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:44.126566 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-89b4bf89b-sxltm" event={"ID":"5b2875d4-ab7f-4bf1-851d-a4006df17049","Type":"ContainerStarted","Data":"7b9c31b501b845c1fe7b49cc83f0aae66415bdf7741b0e64b940ea5e7ba7d882"} Jul 06 19:39:44.126706 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:44.126678 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-89b4bf89b-sxltm" Jul 06 19:39:44.150854 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:44.150813 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-89b4bf89b-sxltm" podStartSLOduration=1.607830909 podStartE2EDuration="2.150801099s" totalImagesPullingTime="542.97019ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:39:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:39:44.148879424 +0000 UTC m=+792.968968129" watchObservedRunningTime="2026-07-06 19:39:44.150801099 +0000 UTC m=+792.970889808" Jul 06 19:39:55.139746 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:39:55.139657 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-89b4bf89b-sxltm" Jul 06 19:40:02.178439 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:40:02.178387 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-69ff557b65-zcl5k" podUID="21b51266-ec66-4a9c-aed6-f25fa6b55785" containerName="console" containerID="cri-o://aa48bcb1196af679d074d1f1e6058b85623ef2041415a6350be2017662d84147" gracePeriod=15 Jul 06 19:40:02.437495 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:40:02.437444 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69ff557b65-zcl5k" Jul 06 19:40:02.612036 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:40:02.612003 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/21b51266-ec66-4a9c-aed6-f25fa6b55785-console-oauth-config\") pod \"21b51266-ec66-4a9c-aed6-f25fa6b55785\" (UID: \"21b51266-ec66-4a9c-aed6-f25fa6b55785\") " Jul 06 19:40:02.612036 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:40:02.612037 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pw87d\" (UniqueName: \"kubernetes.io/projected/21b51266-ec66-4a9c-aed6-f25fa6b55785-kube-api-access-pw87d\") pod \"21b51266-ec66-4a9c-aed6-f25fa6b55785\" (UID: \"21b51266-ec66-4a9c-aed6-f25fa6b55785\") " Jul 06 19:40:02.612277 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:40:02.612080 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/21b51266-ec66-4a9c-aed6-f25fa6b55785-oauth-serving-cert\") pod \"21b51266-ec66-4a9c-aed6-f25fa6b55785\" (UID: \"21b51266-ec66-4a9c-aed6-f25fa6b55785\") " Jul 06 19:40:02.612277 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:40:02.612125 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/21b51266-ec66-4a9c-aed6-f25fa6b55785-console-config\") pod \"21b51266-ec66-4a9c-aed6-f25fa6b55785\" (UID: \"21b51266-ec66-4a9c-aed6-f25fa6b55785\") " Jul 06 19:40:02.612277 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:40:02.612149 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21b51266-ec66-4a9c-aed6-f25fa6b55785-trusted-ca-bundle\") pod \"21b51266-ec66-4a9c-aed6-f25fa6b55785\" (UID: \"21b51266-ec66-4a9c-aed6-f25fa6b55785\") " Jul 06 19:40:02.612277 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:40:02.612218 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/21b51266-ec66-4a9c-aed6-f25fa6b55785-console-serving-cert\") pod \"21b51266-ec66-4a9c-aed6-f25fa6b55785\" (UID: \"21b51266-ec66-4a9c-aed6-f25fa6b55785\") " Jul 06 19:40:02.612277 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:40:02.612263 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/21b51266-ec66-4a9c-aed6-f25fa6b55785-service-ca\") pod \"21b51266-ec66-4a9c-aed6-f25fa6b55785\" (UID: \"21b51266-ec66-4a9c-aed6-f25fa6b55785\") " Jul 06 19:40:02.612542 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:40:02.612501 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21b51266-ec66-4a9c-aed6-f25fa6b55785-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "21b51266-ec66-4a9c-aed6-f25fa6b55785" (UID: "21b51266-ec66-4a9c-aed6-f25fa6b55785"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:40:02.612599 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:40:02.612560 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21b51266-ec66-4a9c-aed6-f25fa6b55785-console-config" (OuterVolumeSpecName: "console-config") pod "21b51266-ec66-4a9c-aed6-f25fa6b55785" (UID: "21b51266-ec66-4a9c-aed6-f25fa6b55785"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:40:02.612651 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:40:02.612611 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21b51266-ec66-4a9c-aed6-f25fa6b55785-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "21b51266-ec66-4a9c-aed6-f25fa6b55785" (UID: "21b51266-ec66-4a9c-aed6-f25fa6b55785"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:40:02.612699 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:40:02.612676 2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/21b51266-ec66-4a9c-aed6-f25fa6b55785-oauth-serving-cert\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:40:02.612699 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:40:02.612692 2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/21b51266-ec66-4a9c-aed6-f25fa6b55785-console-config\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:40:02.612788 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:40:02.612747 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21b51266-ec66-4a9c-aed6-f25fa6b55785-service-ca" (OuterVolumeSpecName: "service-ca") pod "21b51266-ec66-4a9c-aed6-f25fa6b55785" (UID: "21b51266-ec66-4a9c-aed6-f25fa6b55785"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:40:02.614155 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:40:02.614124 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21b51266-ec66-4a9c-aed6-f25fa6b55785-kube-api-access-pw87d" (OuterVolumeSpecName: "kube-api-access-pw87d") pod "21b51266-ec66-4a9c-aed6-f25fa6b55785" (UID: "21b51266-ec66-4a9c-aed6-f25fa6b55785"). InnerVolumeSpecName "kube-api-access-pw87d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:40:02.614497 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:40:02.614466 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21b51266-ec66-4a9c-aed6-f25fa6b55785-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "21b51266-ec66-4a9c-aed6-f25fa6b55785" (UID: "21b51266-ec66-4a9c-aed6-f25fa6b55785"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:40:02.614588 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:40:02.614475 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21b51266-ec66-4a9c-aed6-f25fa6b55785-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "21b51266-ec66-4a9c-aed6-f25fa6b55785" (UID: "21b51266-ec66-4a9c-aed6-f25fa6b55785"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:40:02.713728 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:40:02.713670 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21b51266-ec66-4a9c-aed6-f25fa6b55785-trusted-ca-bundle\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:40:02.713728 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:40:02.713691 2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/21b51266-ec66-4a9c-aed6-f25fa6b55785-console-serving-cert\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:40:02.713728 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:40:02.713700 2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/21b51266-ec66-4a9c-aed6-f25fa6b55785-service-ca\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:40:02.713728 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:40:02.713708 2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/21b51266-ec66-4a9c-aed6-f25fa6b55785-console-oauth-config\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:40:02.713728 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:40:02.713716 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw87d\" (UniqueName: \"kubernetes.io/projected/21b51266-ec66-4a9c-aed6-f25fa6b55785-kube-api-access-pw87d\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:40:03.243419 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:40:03.243386 2575 generic.go:356] "Generic (PLEG): container finished" podID="21b51266-ec66-4a9c-aed6-f25fa6b55785" containerID="aa48bcb1196af679d074d1f1e6058b85623ef2041415a6350be2017662d84147" exitCode=0 Jul 06 19:40:03.243874 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:40:03.243438 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69ff557b65-zcl5k" Jul 06 19:40:03.243874 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:40:03.243438 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69ff557b65-zcl5k" event={"ID":"21b51266-ec66-4a9c-aed6-f25fa6b55785","Type":"ContainerDied","Data":"aa48bcb1196af679d074d1f1e6058b85623ef2041415a6350be2017662d84147"} Jul 06 19:40:03.243874 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:40:03.243566 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69ff557b65-zcl5k" event={"ID":"21b51266-ec66-4a9c-aed6-f25fa6b55785","Type":"ContainerDied","Data":"af827904148dc7a67e55eb13075dc80df77febee2e9a7d416583a185297efb61"} Jul 06 19:40:03.243874 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:40:03.243591 2575 scope.go:117] "RemoveContainer" containerID="aa48bcb1196af679d074d1f1e6058b85623ef2041415a6350be2017662d84147" Jul 06 19:40:03.253469 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:40:03.253451 2575 scope.go:117] "RemoveContainer" containerID="aa48bcb1196af679d074d1f1e6058b85623ef2041415a6350be2017662d84147" Jul 06 19:40:03.253721 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:40:03.253700 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa48bcb1196af679d074d1f1e6058b85623ef2041415a6350be2017662d84147\": container with ID starting with aa48bcb1196af679d074d1f1e6058b85623ef2041415a6350be2017662d84147 not found: ID does not exist" containerID="aa48bcb1196af679d074d1f1e6058b85623ef2041415a6350be2017662d84147" Jul 06 19:40:03.253820 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:40:03.253730 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa48bcb1196af679d074d1f1e6058b85623ef2041415a6350be2017662d84147"} err="failed to get container status \"aa48bcb1196af679d074d1f1e6058b85623ef2041415a6350be2017662d84147\": rpc error: code = NotFound desc = could not find container \"aa48bcb1196af679d074d1f1e6058b85623ef2041415a6350be2017662d84147\": container with ID starting with aa48bcb1196af679d074d1f1e6058b85623ef2041415a6350be2017662d84147 not found: ID does not exist" Jul 06 19:40:03.269839 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:40:03.269816 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69ff557b65-zcl5k"] Jul 06 19:40:03.288880 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:40:03.288860 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-69ff557b65-zcl5k"] Jul 06 19:40:03.794724 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:40:03.794697 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21b51266-ec66-4a9c-aed6-f25fa6b55785" path="/var/lib/kubelet/pods/21b51266-ec66-4a9c-aed6-f25fa6b55785/volumes" Jul 06 19:41:26.134259 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:26.134217 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-89b4bf89b-sxltm" podUID="5b2875d4-ab7f-4bf1-851d-a4006df17049" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.64:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 06 19:41:26.757611 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:26.757576 2575 generic.go:356] "Generic (PLEG): container finished" podID="ebc5be6f-3e75-4208-b275-a4752dbf5b06" containerID="a30df086f307a4db8b7b25e9fc01dde496d7757508200ff08407de5687d77997" exitCode=1 Jul 06 19:41:26.757764 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:26.757654 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5cc9j" event={"ID":"ebc5be6f-3e75-4208-b275-a4752dbf5b06","Type":"ContainerDied","Data":"a30df086f307a4db8b7b25e9fc01dde496d7757508200ff08407de5687d77997"} Jul 06 19:41:26.757993 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:26.757978 2575 scope.go:117] "RemoveContainer" containerID="a30df086f307a4db8b7b25e9fc01dde496d7757508200ff08407de5687d77997" Jul 06 19:41:27.766442 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:27.766391 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5cc9j" event={"ID":"ebc5be6f-3e75-4208-b275-a4752dbf5b06","Type":"ContainerStarted","Data":"3965cea53459f943501418f0c645e461438d83268cba94a0027b1d7b427495e1"} Jul 06 19:41:27.766874 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:27.766656 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5cc9j" Jul 06 19:41:30.577578 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:30.577545 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8vh5h" podUID="1c159251-98a2-4993-af7d-aecf281fbf67" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.39:8081/readyz\": dial tcp 10.134.0.39:8081: connect: connection refused" Jul 06 19:41:30.762701 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:30.762670 2575 prober.go:120] "Probe failed" probeType="Liveness" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8xjts" podUID="e692ea80-46aa-4916-a536-ce492c375f61" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.46:8081/healthz\": dial tcp 10.134.0.46:8081: connect: connection refused" Jul 06 19:41:30.788380 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:30.788355 2575 generic.go:356] "Generic (PLEG): container finished" podID="1c159251-98a2-4993-af7d-aecf281fbf67" containerID="8e85871941e3b1613d6a2d7677f946e788137fd3f1e2f99a6fc1d20117458f49" exitCode=1 Jul 06 19:41:30.788486 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:30.788435 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8vh5h" event={"ID":"1c159251-98a2-4993-af7d-aecf281fbf67","Type":"ContainerDied","Data":"8e85871941e3b1613d6a2d7677f946e788137fd3f1e2f99a6fc1d20117458f49"} Jul 06 19:41:30.788717 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:30.788702 2575 scope.go:117] "RemoveContainer" containerID="8e85871941e3b1613d6a2d7677f946e788137fd3f1e2f99a6fc1d20117458f49" Jul 06 19:41:30.790089 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:30.790068 2575 generic.go:356] "Generic (PLEG): container finished" podID="bba14ce7-8926-47cd-bf1c-474c7d179cdf" containerID="798211a939e8f4b52490eee5b99ab340b10db5f1028af2a9921ab5439f36076e" exitCode=1 Jul 06 19:41:30.790170 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:30.790142 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-qc9r4" event={"ID":"bba14ce7-8926-47cd-bf1c-474c7d179cdf","Type":"ContainerDied","Data":"798211a939e8f4b52490eee5b99ab340b10db5f1028af2a9921ab5439f36076e"} Jul 06 19:41:30.790436 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:30.790421 2575 scope.go:117] "RemoveContainer" containerID="798211a939e8f4b52490eee5b99ab340b10db5f1028af2a9921ab5439f36076e" Jul 06 19:41:30.791879 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:30.791843 2575 generic.go:356] "Generic (PLEG): container finished" podID="e692ea80-46aa-4916-a536-ce492c375f61" containerID="0dee66a9400007351fe5737d7484137056763fb8e2ac7b6dcb4bf6bee1259338" exitCode=1 Jul 06 19:41:30.791959 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:30.791922 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8xjts" event={"ID":"e692ea80-46aa-4916-a536-ce492c375f61","Type":"ContainerDied","Data":"0dee66a9400007351fe5737d7484137056763fb8e2ac7b6dcb4bf6bee1259338"} Jul 06 19:41:30.792232 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:30.792213 2575 scope.go:117] "RemoveContainer" containerID="0dee66a9400007351fe5737d7484137056763fb8e2ac7b6dcb4bf6bee1259338" Jul 06 19:41:30.793608 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:30.793592 2575 generic.go:356] "Generic (PLEG): container finished" podID="5b2875d4-ab7f-4bf1-851d-a4006df17049" containerID="37125d68f3196d24d852c7114858b7634435c3b85f41caf4acd0e13fcd2e2b32" exitCode=1 Jul 06 19:41:30.793711 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:30.793677 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-89b4bf89b-sxltm" event={"ID":"5b2875d4-ab7f-4bf1-851d-a4006df17049","Type":"ContainerDied","Data":"37125d68f3196d24d852c7114858b7634435c3b85f41caf4acd0e13fcd2e2b32"} Jul 06 19:41:30.794030 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:30.794010 2575 scope.go:117] "RemoveContainer" containerID="37125d68f3196d24d852c7114858b7634435c3b85f41caf4acd0e13fcd2e2b32" Jul 06 19:41:30.795499 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:30.795476 2575 generic.go:356] "Generic (PLEG): container finished" podID="3a81b584-f08e-4e8d-922c-5737522ef685" containerID="fae69ce0e23e71de2593c5fd187efc580b03a62e5117330f38eefeb64d58abf6" exitCode=1 Jul 06 19:41:30.795630 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:30.795516 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x88hr" event={"ID":"3a81b584-f08e-4e8d-922c-5737522ef685","Type":"ContainerDied","Data":"fae69ce0e23e71de2593c5fd187efc580b03a62e5117330f38eefeb64d58abf6"} Jul 06 19:41:30.795846 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:30.795827 2575 scope.go:117] "RemoveContainer" containerID="fae69ce0e23e71de2593c5fd187efc580b03a62e5117330f38eefeb64d58abf6" Jul 06 19:41:30.796103 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:30.796082 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 06 19:41:30.797707 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:30.797690 2575 generic.go:356] "Generic (PLEG): container finished" podID="70853bf3-11eb-4d4b-8b27-e5be7b2dfe97" containerID="869f462b48133f29b561b241eef491fca347a30bf1d839c8ea77791d0964b057" exitCode=0 Jul 06 19:41:30.797784 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:30.797759 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gg299" event={"ID":"70853bf3-11eb-4d4b-8b27-e5be7b2dfe97","Type":"ContainerDied","Data":"869f462b48133f29b561b241eef491fca347a30bf1d839c8ea77791d0964b057"} Jul 06 19:41:30.798011 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:30.797997 2575 scope.go:117] "RemoveContainer" containerID="869f462b48133f29b561b241eef491fca347a30bf1d839c8ea77791d0964b057" Jul 06 19:41:30.800428 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:30.800410 2575 generic.go:356] "Generic (PLEG): container finished" podID="de811e8a-5b85-4707-abe7-79107ab0a37a" containerID="2a93ab5ba80dc4bcff47229dd64bf581d0e0f515cc52aeb9860a4a2c8f584d9d" exitCode=1 Jul 06 19:41:30.800543 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:30.800485 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-l267l" event={"ID":"de811e8a-5b85-4707-abe7-79107ab0a37a","Type":"ContainerDied","Data":"2a93ab5ba80dc4bcff47229dd64bf581d0e0f515cc52aeb9860a4a2c8f584d9d"} Jul 06 19:41:30.800933 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:30.800911 2575 scope.go:117] "RemoveContainer" containerID="2a93ab5ba80dc4bcff47229dd64bf581d0e0f515cc52aeb9860a4a2c8f584d9d" Jul 06 19:41:30.806646 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:30.806622 2575 generic.go:356] "Generic (PLEG): container finished" podID="67cd03db-d916-4703-9f42-b5e0df241829" containerID="ce6539d177f1f11c19f527195e4576218a95a3ce9e5647e186c0590c4d312081" exitCode=1 Jul 06 19:41:30.806742 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:30.806654 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-l8hsx" event={"ID":"67cd03db-d916-4703-9f42-b5e0df241829","Type":"ContainerDied","Data":"ce6539d177f1f11c19f527195e4576218a95a3ce9e5647e186c0590c4d312081"} Jul 06 19:41:30.806974 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:30.806960 2575 scope.go:117] "RemoveContainer" containerID="ce6539d177f1f11c19f527195e4576218a95a3ce9e5647e186c0590c4d312081" Jul 06 19:41:31.814083 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:31.814050 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-l8hsx" event={"ID":"67cd03db-d916-4703-9f42-b5e0df241829","Type":"ContainerStarted","Data":"cc819a28f6057a0a6a61017c5cd39efd8a7fe2752b06ee8a6a8ae5b67ea0a933"} Jul 06 19:41:31.814503 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:31.814315 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-l8hsx" Jul 06 19:41:31.815716 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:31.815695 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8vh5h" event={"ID":"1c159251-98a2-4993-af7d-aecf281fbf67","Type":"ContainerStarted","Data":"4a08ddb2e3a2fc68ed8894e44d962139e2de20063706be3c7cfb116450c0c9ea"} Jul 06 19:41:31.815893 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:31.815878 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8vh5h" Jul 06 19:41:31.817324 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:31.817306 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-qc9r4" event={"ID":"bba14ce7-8926-47cd-bf1c-474c7d179cdf","Type":"ContainerStarted","Data":"6aa9d08a60240957bebef42b54fd2fed8d49c10491f0972de25185bab8dc248c"} Jul 06 19:41:31.817488 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:31.817474 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-qc9r4" Jul 06 19:41:31.818983 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:31.818965 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8xjts" event={"ID":"e692ea80-46aa-4916-a536-ce492c375f61","Type":"ContainerStarted","Data":"1d2e422088619ea6249cb32e2b56a83c291b0b3454c784ec0ebcbf1ca6b5976c"} Jul 06 19:41:31.819175 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:31.819162 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8xjts" Jul 06 19:41:31.820631 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:31.820613 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-89b4bf89b-sxltm" event={"ID":"5b2875d4-ab7f-4bf1-851d-a4006df17049","Type":"ContainerStarted","Data":"acf72047813ced24cf862539ac3d6f8df2dcbfda185d6a5da6f4912332cec44d"} Jul 06 19:41:31.820759 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:31.820747 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-89b4bf89b-sxltm" Jul 06 19:41:31.822333 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:31.822309 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x88hr" event={"ID":"3a81b584-f08e-4e8d-922c-5737522ef685","Type":"ContainerStarted","Data":"fa021a98356da153910b2e21efa21c1085b8dba8b02515ce66ed689d52af7dba"} Jul 06 19:41:31.822471 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:31.822459 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x88hr" Jul 06 19:41:31.824046 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:31.824030 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gg299" event={"ID":"70853bf3-11eb-4d4b-8b27-e5be7b2dfe97","Type":"ContainerStarted","Data":"3781b3e34b2ae88c04897908dcf1f9313bbfad6b4e91a14ed95c95b041e6c656"} Jul 06 19:41:31.824233 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:31.824221 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gg299" Jul 06 19:41:31.826044 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:31.826027 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-l267l" event={"ID":"de811e8a-5b85-4707-abe7-79107ab0a37a","Type":"ContainerStarted","Data":"08b9b95692999b2f000bc5822d24c64498420209e7a0428abdfcecd847462361"} Jul 06 19:41:31.826209 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:31.826196 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-l267l" Jul 06 19:41:38.774913 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:38.774877 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5cc9j" Jul 06 19:41:42.835071 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:42.835028 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gg299" Jul 06 19:41:42.835602 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:42.835129 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-qc9r4" Jul 06 19:41:42.835602 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:42.835176 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x88hr" Jul 06 19:41:42.835800 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:42.835781 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-l267l" Jul 06 19:41:42.835867 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:42.835859 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8xjts" Jul 06 19:41:42.841892 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:41:42.841866 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-89b4bf89b-sxltm" Jul 06 19:42:02.835009 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:42:02.834976 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8vh5h" Jul 06 19:42:02.838510 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:42:02.838488 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-l8hsx" Jul 06 19:44:41.184289 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:44:41.184207 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gg299"] Jul 06 19:44:41.184834 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:44:41.184457 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gg299" podUID="70853bf3-11eb-4d4b-8b27-e5be7b2dfe97" containerName="manager" containerID="cri-o://3781b3e34b2ae88c04897908dcf1f9313bbfad6b4e91a14ed95c95b041e6c656" gracePeriod=10 Jul 06 19:44:42.740070 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:44:42.740043 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gg299" Jul 06 19:44:42.875132 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:44:42.875110 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gscfs\" (UniqueName: \"kubernetes.io/projected/70853bf3-11eb-4d4b-8b27-e5be7b2dfe97-kube-api-access-gscfs\") pod \"70853bf3-11eb-4d4b-8b27-e5be7b2dfe97\" (UID: \"70853bf3-11eb-4d4b-8b27-e5be7b2dfe97\") " Jul 06 19:44:42.875313 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:44:42.875204 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70853bf3-11eb-4d4b-8b27-e5be7b2dfe97-extensions-socket-volume\") pod \"70853bf3-11eb-4d4b-8b27-e5be7b2dfe97\" (UID: \"70853bf3-11eb-4d4b-8b27-e5be7b2dfe97\") " Jul 06 19:44:42.875563 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:44:42.875537 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70853bf3-11eb-4d4b-8b27-e5be7b2dfe97-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "70853bf3-11eb-4d4b-8b27-e5be7b2dfe97" (UID: "70853bf3-11eb-4d4b-8b27-e5be7b2dfe97"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:44:42.877024 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:44:42.877002 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70853bf3-11eb-4d4b-8b27-e5be7b2dfe97-kube-api-access-gscfs" (OuterVolumeSpecName: "kube-api-access-gscfs") pod "70853bf3-11eb-4d4b-8b27-e5be7b2dfe97" (UID: "70853bf3-11eb-4d4b-8b27-e5be7b2dfe97"). InnerVolumeSpecName "kube-api-access-gscfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:44:42.976299 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:44:42.976273 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gscfs\" (UniqueName: \"kubernetes.io/projected/70853bf3-11eb-4d4b-8b27-e5be7b2dfe97-kube-api-access-gscfs\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:44:42.976299 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:44:42.976298 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70853bf3-11eb-4d4b-8b27-e5be7b2dfe97-extensions-socket-volume\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:44:42.987383 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:44:42.987354 2575 generic.go:356] "Generic (PLEG): container finished" podID="70853bf3-11eb-4d4b-8b27-e5be7b2dfe97" containerID="3781b3e34b2ae88c04897908dcf1f9313bbfad6b4e91a14ed95c95b041e6c656" exitCode=0 Jul 06 19:44:42.987473 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:44:42.987385 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gg299" event={"ID":"70853bf3-11eb-4d4b-8b27-e5be7b2dfe97","Type":"ContainerDied","Data":"3781b3e34b2ae88c04897908dcf1f9313bbfad6b4e91a14ed95c95b041e6c656"} Jul 06 19:44:42.987473 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:44:42.987402 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gg299" Jul 06 19:44:42.987473 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:44:42.987439 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gg299" event={"ID":"70853bf3-11eb-4d4b-8b27-e5be7b2dfe97","Type":"ContainerDied","Data":"935098efae8dbc218ff449c6a685f0bc40055dba351f1da757f8e5b68b3c01df"} Jul 06 19:44:42.987473 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:44:42.987463 2575 scope.go:117] "RemoveContainer" containerID="3781b3e34b2ae88c04897908dcf1f9313bbfad6b4e91a14ed95c95b041e6c656" Jul 06 19:44:42.998034 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:44:42.998017 2575 scope.go:117] "RemoveContainer" containerID="869f462b48133f29b561b241eef491fca347a30bf1d839c8ea77791d0964b057" Jul 06 19:44:43.007231 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:44:43.007215 2575 scope.go:117] "RemoveContainer" containerID="3781b3e34b2ae88c04897908dcf1f9313bbfad6b4e91a14ed95c95b041e6c656" Jul 06 19:44:43.007502 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:44:43.007482 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3781b3e34b2ae88c04897908dcf1f9313bbfad6b4e91a14ed95c95b041e6c656\": container with ID starting with 3781b3e34b2ae88c04897908dcf1f9313bbfad6b4e91a14ed95c95b041e6c656 not found: ID does not exist" containerID="3781b3e34b2ae88c04897908dcf1f9313bbfad6b4e91a14ed95c95b041e6c656" Jul 06 19:44:43.007566 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:44:43.007509 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3781b3e34b2ae88c04897908dcf1f9313bbfad6b4e91a14ed95c95b041e6c656"} err="failed to get container status \"3781b3e34b2ae88c04897908dcf1f9313bbfad6b4e91a14ed95c95b041e6c656\": rpc error: code = NotFound desc = could not find container \"3781b3e34b2ae88c04897908dcf1f9313bbfad6b4e91a14ed95c95b041e6c656\": container with ID starting with 3781b3e34b2ae88c04897908dcf1f9313bbfad6b4e91a14ed95c95b041e6c656 not found: ID does not exist" Jul 06 19:44:43.007566 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:44:43.007525 2575 scope.go:117] "RemoveContainer" containerID="869f462b48133f29b561b241eef491fca347a30bf1d839c8ea77791d0964b057" Jul 06 19:44:43.007760 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:44:43.007745 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"869f462b48133f29b561b241eef491fca347a30bf1d839c8ea77791d0964b057\": container with ID starting with 869f462b48133f29b561b241eef491fca347a30bf1d839c8ea77791d0964b057 not found: ID does not exist" containerID="869f462b48133f29b561b241eef491fca347a30bf1d839c8ea77791d0964b057" Jul 06 19:44:43.007800 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:44:43.007764 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"869f462b48133f29b561b241eef491fca347a30bf1d839c8ea77791d0964b057"} err="failed to get container status \"869f462b48133f29b561b241eef491fca347a30bf1d839c8ea77791d0964b057\": rpc error: code = NotFound desc = could not find container \"869f462b48133f29b561b241eef491fca347a30bf1d839c8ea77791d0964b057\": container with ID starting with 869f462b48133f29b561b241eef491fca347a30bf1d839c8ea77791d0964b057 not found: ID does not exist" Jul 06 19:44:43.012558 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:44:43.012538 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gg299"] Jul 06 19:44:43.017767 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:44:43.017740 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gg299"] Jul 06 19:44:43.795669 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:44:43.795639 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70853bf3-11eb-4d4b-8b27-e5be7b2dfe97" path="/var/lib/kubelet/pods/70853bf3-11eb-4d4b-8b27-e5be7b2dfe97/volumes" Jul 06 19:45:00.177506 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:00.177469 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29722785-t92np"] Jul 06 19:45:00.177991 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:00.177974 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70853bf3-11eb-4d4b-8b27-e5be7b2dfe97" containerName="manager" Jul 06 19:45:00.178060 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:00.177993 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="70853bf3-11eb-4d4b-8b27-e5be7b2dfe97" containerName="manager" Jul 06 19:45:00.178060 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:00.178001 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70853bf3-11eb-4d4b-8b27-e5be7b2dfe97" containerName="manager" Jul 06 19:45:00.178060 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:00.178007 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="70853bf3-11eb-4d4b-8b27-e5be7b2dfe97" containerName="manager" Jul 06 19:45:00.178060 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:00.178021 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21b51266-ec66-4a9c-aed6-f25fa6b55785" containerName="console" Jul 06 19:45:00.178060 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:00.178027 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="21b51266-ec66-4a9c-aed6-f25fa6b55785" containerName="console" Jul 06 19:45:00.178222 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:00.178101 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="70853bf3-11eb-4d4b-8b27-e5be7b2dfe97" containerName="manager" Jul 06 19:45:00.178222 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:00.178111 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="21b51266-ec66-4a9c-aed6-f25fa6b55785" containerName="console" Jul 06 19:45:00.181300 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:00.181283 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722785-t92np" Jul 06 19:45:00.184242 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:00.184218 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-q8w7d\"" Jul 06 19:45:00.190477 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:00.190454 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722785-t92np"] Jul 06 19:45:00.324796 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:00.324770 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5qwd\" (UniqueName: \"kubernetes.io/projected/bc158d81-1ccd-4e69-b620-0cb53d5c98ec-kube-api-access-n5qwd\") pod \"maas-api-key-cleanup-29722785-t92np\" (UID: \"bc158d81-1ccd-4e69-b620-0cb53d5c98ec\") " pod="opendatahub/maas-api-key-cleanup-29722785-t92np" Jul 06 19:45:00.426080 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:00.426055 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5qwd\" (UniqueName: \"kubernetes.io/projected/bc158d81-1ccd-4e69-b620-0cb53d5c98ec-kube-api-access-n5qwd\") pod \"maas-api-key-cleanup-29722785-t92np\" (UID: \"bc158d81-1ccd-4e69-b620-0cb53d5c98ec\") " pod="opendatahub/maas-api-key-cleanup-29722785-t92np" Jul 06 19:45:00.436155 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:00.436104 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5qwd\" (UniqueName: \"kubernetes.io/projected/bc158d81-1ccd-4e69-b620-0cb53d5c98ec-kube-api-access-n5qwd\") pod \"maas-api-key-cleanup-29722785-t92np\" (UID: \"bc158d81-1ccd-4e69-b620-0cb53d5c98ec\") " pod="opendatahub/maas-api-key-cleanup-29722785-t92np" Jul 06 19:45:00.493473 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:00.493443 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722785-t92np" Jul 06 19:45:00.636990 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:00.636965 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722785-t92np"] Jul 06 19:45:00.637575 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:45:00.637549 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc158d81_1ccd_4e69_b620_0cb53d5c98ec.slice/crio-a2d3ddeacebf5a9e6668b688fd626671004d8b249c9d50cc8152fe795d1050e9 WatchSource:0}: Error finding container a2d3ddeacebf5a9e6668b688fd626671004d8b249c9d50cc8152fe795d1050e9: Status 404 returned error can't find the container with id a2d3ddeacebf5a9e6668b688fd626671004d8b249c9d50cc8152fe795d1050e9 Jul 06 19:45:00.639847 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:00.639830 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:45:01.101623 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:01.101582 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722785-t92np" event={"ID":"bc158d81-1ccd-4e69-b620-0cb53d5c98ec","Type":"ContainerStarted","Data":"a2d3ddeacebf5a9e6668b688fd626671004d8b249c9d50cc8152fe795d1050e9"} Jul 06 19:45:14.183026 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:14.182999 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722785-t92np" event={"ID":"bc158d81-1ccd-4e69-b620-0cb53d5c98ec","Type":"ContainerStarted","Data":"358085d14368a1fb5c3b4df9a708a5c0e0acbfe304cc7e2bc56e8b3812851341"} Jul 06 19:45:14.204989 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:14.204945 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29722785-t92np" podStartSLOduration=1.139635515 podStartE2EDuration="14.204929964s" totalImagesPullingTime="13.065294449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:45:14.202370054 +0000 UTC m=+1123.022458778" watchObservedRunningTime="2026-07-06 19:45:14.204929964 +0000 UTC m=+1123.025018673" Jul 06 19:45:15.191723 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:15.191695 2575 generic.go:356] "Generic (PLEG): container finished" podID="bc158d81-1ccd-4e69-b620-0cb53d5c98ec" containerID="358085d14368a1fb5c3b4df9a708a5c0e0acbfe304cc7e2bc56e8b3812851341" exitCode=0 Jul 06 19:45:15.192044 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:15.191785 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722785-t92np" event={"ID":"bc158d81-1ccd-4e69-b620-0cb53d5c98ec","Type":"ContainerDied","Data":"358085d14368a1fb5c3b4df9a708a5c0e0acbfe304cc7e2bc56e8b3812851341"} Jul 06 19:45:16.338119 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:16.338095 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722785-t92np" Jul 06 19:45:16.452279 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:16.452209 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5qwd\" (UniqueName: \"kubernetes.io/projected/bc158d81-1ccd-4e69-b620-0cb53d5c98ec-kube-api-access-n5qwd\") pod \"bc158d81-1ccd-4e69-b620-0cb53d5c98ec\" (UID: \"bc158d81-1ccd-4e69-b620-0cb53d5c98ec\") " Jul 06 19:45:16.454067 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:16.454043 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc158d81-1ccd-4e69-b620-0cb53d5c98ec-kube-api-access-n5qwd" (OuterVolumeSpecName: "kube-api-access-n5qwd") pod "bc158d81-1ccd-4e69-b620-0cb53d5c98ec" (UID: "bc158d81-1ccd-4e69-b620-0cb53d5c98ec"). InnerVolumeSpecName "kube-api-access-n5qwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:45:16.553428 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:16.553404 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5qwd\" (UniqueName: \"kubernetes.io/projected/bc158d81-1ccd-4e69-b620-0cb53d5c98ec-kube-api-access-n5qwd\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:45:17.205833 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:17.205799 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722785-t92np" Jul 06 19:45:17.205989 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:17.205838 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722785-t92np" event={"ID":"bc158d81-1ccd-4e69-b620-0cb53d5c98ec","Type":"ContainerDied","Data":"a2d3ddeacebf5a9e6668b688fd626671004d8b249c9d50cc8152fe795d1050e9"} Jul 06 19:45:17.205989 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:17.205916 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2d3ddeacebf5a9e6668b688fd626671004d8b249c9d50cc8152fe795d1050e9" Jul 06 19:45:43.835142 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:43.835071 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-89b4bf89b-sxltm" podUID="5b2875d4-ab7f-4bf1-851d-a4006df17049" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.64:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 06 19:45:50.426260 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:50.426229 2575 generic.go:356] "Generic (PLEG): container finished" podID="e692ea80-46aa-4916-a536-ce492c375f61" containerID="1d2e422088619ea6249cb32e2b56a83c291b0b3454c784ec0ebcbf1ca6b5976c" exitCode=1 Jul 06 19:45:50.426714 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:50.426300 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8xjts" event={"ID":"e692ea80-46aa-4916-a536-ce492c375f61","Type":"ContainerDied","Data":"1d2e422088619ea6249cb32e2b56a83c291b0b3454c784ec0ebcbf1ca6b5976c"} Jul 06 19:45:50.426714 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:50.426357 2575 scope.go:117] "RemoveContainer" containerID="0dee66a9400007351fe5737d7484137056763fb8e2ac7b6dcb4bf6bee1259338" Jul 06 19:45:50.426714 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:50.426600 2575 scope.go:117] "RemoveContainer" containerID="1d2e422088619ea6249cb32e2b56a83c291b0b3454c784ec0ebcbf1ca6b5976c" Jul 06 19:45:50.426845 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:45:50.426817 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=dns-operator-controller-manager-864dd78c4c-8xjts_kuadrant-system(e692ea80-46aa-4916-a536-ce492c375f61)\"" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8xjts" podUID="e692ea80-46aa-4916-a536-ce492c375f61" Jul 06 19:45:50.614040 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:50.614003 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6qjzf"] Jul 06 19:45:50.614457 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:50.614445 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc158d81-1ccd-4e69-b620-0cb53d5c98ec" containerName="cleanup" Jul 06 19:45:50.614507 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:50.614459 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc158d81-1ccd-4e69-b620-0cb53d5c98ec" containerName="cleanup" Jul 06 19:45:50.614545 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:50.614539 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc158d81-1ccd-4e69-b620-0cb53d5c98ec" containerName="cleanup" Jul 06 19:45:50.614580 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:50.614551 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="70853bf3-11eb-4d4b-8b27-e5be7b2dfe97" containerName="manager" Jul 06 19:45:50.618830 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:50.618816 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6qjzf" Jul 06 19:45:50.621507 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:50.621484 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-xdz2f\"" Jul 06 19:45:50.640038 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:50.640013 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6qjzf"] Jul 06 19:45:50.736068 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:50.736012 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f406c37-7a7b-4662-9c96-f1b16ba3b0fa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-6qjzf\" (UID: \"2f406c37-7a7b-4662-9c96-f1b16ba3b0fa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6qjzf" Jul 06 19:45:50.736068 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:50.736064 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch8wm\" (UniqueName: \"kubernetes.io/projected/2f406c37-7a7b-4662-9c96-f1b16ba3b0fa-kube-api-access-ch8wm\") pod \"kuadrant-operator-controller-manager-86878c769f-6qjzf\" (UID: \"2f406c37-7a7b-4662-9c96-f1b16ba3b0fa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6qjzf" Jul 06 19:45:50.762965 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:50.762941 2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8xjts" Jul 06 19:45:50.837424 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:50.837400 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch8wm\" (UniqueName: \"kubernetes.io/projected/2f406c37-7a7b-4662-9c96-f1b16ba3b0fa-kube-api-access-ch8wm\") pod \"kuadrant-operator-controller-manager-86878c769f-6qjzf\" (UID: \"2f406c37-7a7b-4662-9c96-f1b16ba3b0fa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6qjzf" Jul 06 19:45:50.837530 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:50.837461 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f406c37-7a7b-4662-9c96-f1b16ba3b0fa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-6qjzf\" (UID: \"2f406c37-7a7b-4662-9c96-f1b16ba3b0fa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6qjzf" Jul 06 19:45:50.837786 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:50.837769 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f406c37-7a7b-4662-9c96-f1b16ba3b0fa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-6qjzf\" (UID: \"2f406c37-7a7b-4662-9c96-f1b16ba3b0fa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6qjzf" Jul 06 19:45:50.849725 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:50.849700 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch8wm\" (UniqueName: \"kubernetes.io/projected/2f406c37-7a7b-4662-9c96-f1b16ba3b0fa-kube-api-access-ch8wm\") pod \"kuadrant-operator-controller-manager-86878c769f-6qjzf\" (UID: \"2f406c37-7a7b-4662-9c96-f1b16ba3b0fa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6qjzf" Jul 06 19:45:50.928713 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:50.928689 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6qjzf" Jul 06 19:45:51.068027 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:51.068001 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6qjzf"] Jul 06 19:45:51.069204 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:45:51.069151 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f406c37_7a7b_4662_9c96_f1b16ba3b0fa.slice/crio-3436cc8795ff4b5b63b73af570e82e80f13261b09a066ffcb879d08c95e11f02 WatchSource:0}: Error finding container 3436cc8795ff4b5b63b73af570e82e80f13261b09a066ffcb879d08c95e11f02: Status 404 returned error can't find the container with id 3436cc8795ff4b5b63b73af570e82e80f13261b09a066ffcb879d08c95e11f02 Jul 06 19:45:51.434722 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:51.434703 2575 scope.go:117] "RemoveContainer" containerID="1d2e422088619ea6249cb32e2b56a83c291b0b3454c784ec0ebcbf1ca6b5976c" Jul 06 19:45:51.435164 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:45:51.434914 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=dns-operator-controller-manager-864dd78c4c-8xjts_kuadrant-system(e692ea80-46aa-4916-a536-ce492c375f61)\"" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8xjts" podUID="e692ea80-46aa-4916-a536-ce492c375f61" Jul 06 19:45:51.436367 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:51.436341 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6qjzf" event={"ID":"2f406c37-7a7b-4662-9c96-f1b16ba3b0fa","Type":"ContainerStarted","Data":"2a475d9d8415189ec35d4a485931501960d0a69674855a69cd25b53efae03cd8"} Jul 06 19:45:51.436470 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:51.436392 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6qjzf" event={"ID":"2f406c37-7a7b-4662-9c96-f1b16ba3b0fa","Type":"ContainerStarted","Data":"3436cc8795ff4b5b63b73af570e82e80f13261b09a066ffcb879d08c95e11f02"} Jul 06 19:45:51.436470 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:51.436420 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6qjzf" Jul 06 19:45:51.522167 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:51.522121 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6qjzf" podStartSLOduration=1.52210709 podStartE2EDuration="1.52210709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:45:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:45:51.519334303 +0000 UTC m=+1160.339423034" watchObservedRunningTime="2026-07-06 19:45:51.52210709 +0000 UTC m=+1160.342195799" Jul 06 19:45:52.833947 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:52.833891 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8xjts" Jul 06 19:45:52.834565 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:45:52.834491 2575 scope.go:117] "RemoveContainer" containerID="1d2e422088619ea6249cb32e2b56a83c291b0b3454c784ec0ebcbf1ca6b5976c" Jul 06 19:45:52.834780 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:45:52.834754 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=dns-operator-controller-manager-864dd78c4c-8xjts_kuadrant-system(e692ea80-46aa-4916-a536-ce492c375f61)\"" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8xjts" podUID="e692ea80-46aa-4916-a536-ce492c375f61" Jul 06 19:46:02.446035 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:46:02.446000 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6qjzf" Jul 06 19:46:06.788620 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:46:06.788589 2575 scope.go:117] "RemoveContainer" containerID="1d2e422088619ea6249cb32e2b56a83c291b0b3454c784ec0ebcbf1ca6b5976c" Jul 06 19:46:07.541801 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:46:07.541767 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8xjts" event={"ID":"e692ea80-46aa-4916-a536-ce492c375f61","Type":"ContainerStarted","Data":"30d96030734266fc8f112e4c57cdcd4632c588b747f96974a195dce7f369be08"} Jul 06 19:46:07.541990 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:46:07.541977 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8xjts" Jul 06 19:46:18.550170 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:46:18.550145 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8xjts" Jul 06 19:51:13.835125 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:51:13.835077 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-89b4bf89b-sxltm" podUID="5b2875d4-ab7f-4bf1-851d-a4006df17049" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.64:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 06 19:52:15.875117 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:15.875047 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk"] Jul 06 19:52:15.879034 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:15.879013 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk" Jul 06 19:52:15.881617 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:15.881591 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 06 19:52:15.881730 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:15.881667 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-6acba6da-openshift-default-dockercfg-dt4ht\"" Jul 06 19:52:15.900512 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:15.900492 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk"] Jul 06 19:52:15.930433 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:15.930413 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/635b03b0-5944-4a1c-8986-fb3819397426-istiod-ca-cert\") pod \"e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk\" (UID: \"635b03b0-5944-4a1c-8986-fb3819397426\") " pod="openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk" Jul 06 19:52:15.930557 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:15.930450 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/635b03b0-5944-4a1c-8986-fb3819397426-workload-certs\") pod \"e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk\" (UID: \"635b03b0-5944-4a1c-8986-fb3819397426\") " pod="openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk" Jul 06 19:52:15.930557 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:15.930486 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/635b03b0-5944-4a1c-8986-fb3819397426-credential-socket\") pod \"e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk\" (UID: \"635b03b0-5944-4a1c-8986-fb3819397426\") " pod="openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk" Jul 06 19:52:15.930557 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:15.930531 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/635b03b0-5944-4a1c-8986-fb3819397426-workload-socket\") pod \"e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk\" (UID: \"635b03b0-5944-4a1c-8986-fb3819397426\") " pod="openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk" Jul 06 19:52:15.930557 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:15.930549 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/635b03b0-5944-4a1c-8986-fb3819397426-istio-podinfo\") pod \"e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk\" (UID: \"635b03b0-5944-4a1c-8986-fb3819397426\") " pod="openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk" Jul 06 19:52:15.930761 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:15.930635 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/635b03b0-5944-4a1c-8986-fb3819397426-istio-token\") pod \"e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk\" (UID: \"635b03b0-5944-4a1c-8986-fb3819397426\") " pod="openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk" Jul 06 19:52:15.930761 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:15.930696 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/635b03b0-5944-4a1c-8986-fb3819397426-istio-envoy\") pod \"e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk\" (UID: \"635b03b0-5944-4a1c-8986-fb3819397426\") " pod="openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk" Jul 06 19:52:15.930761 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:15.930719 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/635b03b0-5944-4a1c-8986-fb3819397426-istio-data\") pod \"e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk\" (UID: \"635b03b0-5944-4a1c-8986-fb3819397426\") " pod="openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk" Jul 06 19:52:15.930761 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:15.930752 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qckdl\" (UniqueName: \"kubernetes.io/projected/635b03b0-5944-4a1c-8986-fb3819397426-kube-api-access-qckdl\") pod \"e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk\" (UID: \"635b03b0-5944-4a1c-8986-fb3819397426\") " pod="openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk" Jul 06 19:52:16.031685 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:16.031659 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/635b03b0-5944-4a1c-8986-fb3819397426-credential-socket\") pod \"e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk\" (UID: \"635b03b0-5944-4a1c-8986-fb3819397426\") " pod="openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk" Jul 06 19:52:16.031812 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:16.031707 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/635b03b0-5944-4a1c-8986-fb3819397426-workload-socket\") pod \"e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk\" (UID: \"635b03b0-5944-4a1c-8986-fb3819397426\") " pod="openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk" Jul 06 19:52:16.031812 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:16.031729 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/635b03b0-5944-4a1c-8986-fb3819397426-istio-podinfo\") pod \"e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk\" (UID: \"635b03b0-5944-4a1c-8986-fb3819397426\") " pod="openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk" Jul 06 19:52:16.031812 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:16.031769 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/635b03b0-5944-4a1c-8986-fb3819397426-istio-token\") pod \"e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk\" (UID: \"635b03b0-5944-4a1c-8986-fb3819397426\") " pod="openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk" Jul 06 19:52:16.031919 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:16.031811 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/635b03b0-5944-4a1c-8986-fb3819397426-istio-envoy\") pod \"e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk\" (UID: \"635b03b0-5944-4a1c-8986-fb3819397426\") " pod="openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk" Jul 06 19:52:16.031919 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:16.031831 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/635b03b0-5944-4a1c-8986-fb3819397426-istio-data\") pod \"e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk\" (UID: \"635b03b0-5944-4a1c-8986-fb3819397426\") " pod="openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk" Jul 06 19:52:16.032018 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:16.031977 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qckdl\" (UniqueName: \"kubernetes.io/projected/635b03b0-5944-4a1c-8986-fb3819397426-kube-api-access-qckdl\") pod \"e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk\" (UID: \"635b03b0-5944-4a1c-8986-fb3819397426\") " pod="openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk" Jul 06 19:52:16.032094 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:16.032071 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/635b03b0-5944-4a1c-8986-fb3819397426-credential-socket\") pod \"e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk\" (UID: \"635b03b0-5944-4a1c-8986-fb3819397426\") " pod="openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk" Jul 06 19:52:16.032148 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:16.032124 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/635b03b0-5944-4a1c-8986-fb3819397426-istio-data\") pod \"e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk\" (UID: \"635b03b0-5944-4a1c-8986-fb3819397426\") " pod="openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk" Jul 06 19:52:16.032219 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:16.032084 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/635b03b0-5944-4a1c-8986-fb3819397426-istiod-ca-cert\") pod \"e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk\" (UID: \"635b03b0-5944-4a1c-8986-fb3819397426\") " pod="openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk" Jul 06 19:52:16.032219 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:16.032207 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/635b03b0-5944-4a1c-8986-fb3819397426-workload-certs\") pod \"e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk\" (UID: \"635b03b0-5944-4a1c-8986-fb3819397426\") " pod="openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk" Jul 06 19:52:16.032451 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:16.032427 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/635b03b0-5944-4a1c-8986-fb3819397426-workload-socket\") pod \"e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk\" (UID: \"635b03b0-5944-4a1c-8986-fb3819397426\") " pod="openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk" Jul 06 19:52:16.032521 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:16.032500 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/635b03b0-5944-4a1c-8986-fb3819397426-workload-certs\") pod \"e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk\" (UID: \"635b03b0-5944-4a1c-8986-fb3819397426\") " pod="openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk" Jul 06 19:52:16.032645 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:16.032627 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/635b03b0-5944-4a1c-8986-fb3819397426-istiod-ca-cert\") pod \"e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk\" (UID: \"635b03b0-5944-4a1c-8986-fb3819397426\") " pod="openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk" Jul 06 19:52:16.033984 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:16.033960 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/635b03b0-5944-4a1c-8986-fb3819397426-istio-envoy\") pod \"e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk\" (UID: \"635b03b0-5944-4a1c-8986-fb3819397426\") " pod="openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk" Jul 06 19:52:16.034175 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:16.034158 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/635b03b0-5944-4a1c-8986-fb3819397426-istio-podinfo\") pod \"e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk\" (UID: \"635b03b0-5944-4a1c-8986-fb3819397426\") " pod="openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk" Jul 06 19:52:16.044912 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:16.044890 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/635b03b0-5944-4a1c-8986-fb3819397426-istio-token\") pod \"e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk\" (UID: \"635b03b0-5944-4a1c-8986-fb3819397426\") " pod="openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk" Jul 06 19:52:16.045262 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:16.045247 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qckdl\" (UniqueName: \"kubernetes.io/projected/635b03b0-5944-4a1c-8986-fb3819397426-kube-api-access-qckdl\") pod \"e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk\" (UID: \"635b03b0-5944-4a1c-8986-fb3819397426\") " pod="openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk" Jul 06 19:52:16.191665 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:16.191605 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk" Jul 06 19:52:16.333249 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:16.333226 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk"] Jul 06 19:52:16.333697 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:52:16.333679 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod635b03b0_5944_4a1c_8986_fb3819397426.slice/crio-124928ccf201656c8d9cc9ace250387cab2aacbcf918ee1257fe8b93131373a8 WatchSource:0}: Error finding container 124928ccf201656c8d9cc9ace250387cab2aacbcf918ee1257fe8b93131373a8: Status 404 returned error can't find the container with id 124928ccf201656c8d9cc9ace250387cab2aacbcf918ee1257fe8b93131373a8 Jul 06 19:52:16.335623 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:16.335607 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:52:16.810839 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:16.810804 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk" event={"ID":"635b03b0-5944-4a1c-8986-fb3819397426","Type":"ContainerStarted","Data":"124928ccf201656c8d9cc9ace250387cab2aacbcf918ee1257fe8b93131373a8"} Jul 06 19:52:18.703158 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:18.703126 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 19:52:18.703416 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:18.703208 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 19:52:18.703416 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:18.703235 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 19:52:18.825248 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:18.825212 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk" event={"ID":"635b03b0-5944-4a1c-8986-fb3819397426","Type":"ContainerStarted","Data":"d5426cfc6b6a5a37d568ec732614ea239553d2eb3b182d885374d321938f0b7a"} Jul 06 19:52:18.850878 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:18.850830 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk" podStartSLOduration=1.483772859 podStartE2EDuration="3.850815639s" totalImagesPullingTime="2.36704278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:52:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:52:18.847475195 +0000 UTC m=+1547.667563907" watchObservedRunningTime="2026-07-06 19:52:18.850815639 +0000 UTC m=+1547.670904348" Jul 06 19:52:19.192313 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:19.192279 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk" Jul 06 19:52:19.193905 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:19.193882 2575 patch_prober.go:28] interesting pod/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Jul 06 19:52:19.194032 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:19.193939 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk" podUID="635b03b0-5944-4a1c-8986-fb3819397426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Jul 06 19:52:20.195458 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:20.195430 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk" Jul 06 19:52:20.837530 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:20.837497 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk" Jul 06 19:52:20.838290 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:20.838271 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk" Jul 06 19:52:22.344129 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:22.344100 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk"] Jul 06 19:52:22.848745 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:22.848714 2575 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk" secret="" err="secret \"e2e-ait-6acba6da-openshift-default-dockercfg-dt4ht\" not found" Jul 06 19:52:23.853634 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:23.853597 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk" podUID="635b03b0-5944-4a1c-8986-fb3819397426" containerName="istio-proxy" containerID="cri-o://d5426cfc6b6a5a37d568ec732614ea239553d2eb3b182d885374d321938f0b7a" gracePeriod=30 Jul 06 19:52:27.882529 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:27.882494 2575 generic.go:356] "Generic (PLEG): container finished" podID="635b03b0-5944-4a1c-8986-fb3819397426" containerID="d5426cfc6b6a5a37d568ec732614ea239553d2eb3b182d885374d321938f0b7a" exitCode=0 Jul 06 19:52:27.882855 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:27.882568 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk" event={"ID":"635b03b0-5944-4a1c-8986-fb3819397426","Type":"ContainerDied","Data":"d5426cfc6b6a5a37d568ec732614ea239553d2eb3b182d885374d321938f0b7a"} Jul 06 19:52:28.010857 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:28.010803 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk" Jul 06 19:52:28.150380 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:28.150321 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/635b03b0-5944-4a1c-8986-fb3819397426-istio-podinfo\") pod \"635b03b0-5944-4a1c-8986-fb3819397426\" (UID: \"635b03b0-5944-4a1c-8986-fb3819397426\") " Jul 06 19:52:28.150380 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:28.150350 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/635b03b0-5944-4a1c-8986-fb3819397426-istiod-ca-cert\") pod \"635b03b0-5944-4a1c-8986-fb3819397426\" (UID: \"635b03b0-5944-4a1c-8986-fb3819397426\") " Jul 06 19:52:28.150380 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:28.150379 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/635b03b0-5944-4a1c-8986-fb3819397426-workload-certs\") pod \"635b03b0-5944-4a1c-8986-fb3819397426\" (UID: \"635b03b0-5944-4a1c-8986-fb3819397426\") " Jul 06 19:52:28.150592 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:28.150408 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/635b03b0-5944-4a1c-8986-fb3819397426-credential-socket\") pod \"635b03b0-5944-4a1c-8986-fb3819397426\" (UID: \"635b03b0-5944-4a1c-8986-fb3819397426\") " Jul 06 19:52:28.150592 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:28.150430 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/635b03b0-5944-4a1c-8986-fb3819397426-istio-envoy\") pod \"635b03b0-5944-4a1c-8986-fb3819397426\" (UID: \"635b03b0-5944-4a1c-8986-fb3819397426\") " Jul 06 19:52:28.150592 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:28.150460 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/635b03b0-5944-4a1c-8986-fb3819397426-istio-data\") pod \"635b03b0-5944-4a1c-8986-fb3819397426\" (UID: \"635b03b0-5944-4a1c-8986-fb3819397426\") " Jul 06 19:52:28.150592 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:28.150505 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/635b03b0-5944-4a1c-8986-fb3819397426-workload-socket\") pod \"635b03b0-5944-4a1c-8986-fb3819397426\" (UID: \"635b03b0-5944-4a1c-8986-fb3819397426\") " Jul 06 19:52:28.150592 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:28.150553 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/635b03b0-5944-4a1c-8986-fb3819397426-istio-token\") pod \"635b03b0-5944-4a1c-8986-fb3819397426\" (UID: \"635b03b0-5944-4a1c-8986-fb3819397426\") " Jul 06 19:52:28.150592 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:28.150581 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qckdl\" (UniqueName: \"kubernetes.io/projected/635b03b0-5944-4a1c-8986-fb3819397426-kube-api-access-qckdl\") pod \"635b03b0-5944-4a1c-8986-fb3819397426\" (UID: \"635b03b0-5944-4a1c-8986-fb3819397426\") " Jul 06 19:52:28.150883 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:28.150701 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/635b03b0-5944-4a1c-8986-fb3819397426-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "635b03b0-5944-4a1c-8986-fb3819397426" (UID: "635b03b0-5944-4a1c-8986-fb3819397426"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:52:28.150883 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:28.150721 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/635b03b0-5944-4a1c-8986-fb3819397426-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "635b03b0-5944-4a1c-8986-fb3819397426" (UID: "635b03b0-5944-4a1c-8986-fb3819397426"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:52:28.150996 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:28.150892 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/635b03b0-5944-4a1c-8986-fb3819397426-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "635b03b0-5944-4a1c-8986-fb3819397426" (UID: "635b03b0-5944-4a1c-8986-fb3819397426"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:52:28.150996 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:28.150913 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/635b03b0-5944-4a1c-8986-fb3819397426-istiod-ca-cert\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:52:28.150996 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:28.150932 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/635b03b0-5944-4a1c-8986-fb3819397426-workload-certs\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:52:28.151165 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:28.151036 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/635b03b0-5944-4a1c-8986-fb3819397426-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "635b03b0-5944-4a1c-8986-fb3819397426" (UID: "635b03b0-5944-4a1c-8986-fb3819397426"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:52:28.151756 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:28.151729 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/635b03b0-5944-4a1c-8986-fb3819397426-istio-data" (OuterVolumeSpecName: "istio-data") pod "635b03b0-5944-4a1c-8986-fb3819397426" (UID: "635b03b0-5944-4a1c-8986-fb3819397426"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:52:28.152768 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:28.152741 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/635b03b0-5944-4a1c-8986-fb3819397426-kube-api-access-qckdl" (OuterVolumeSpecName: "kube-api-access-qckdl") pod "635b03b0-5944-4a1c-8986-fb3819397426" (UID: "635b03b0-5944-4a1c-8986-fb3819397426"). InnerVolumeSpecName "kube-api-access-qckdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:52:28.152768 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:28.152744 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/635b03b0-5944-4a1c-8986-fb3819397426-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "635b03b0-5944-4a1c-8986-fb3819397426" (UID: "635b03b0-5944-4a1c-8986-fb3819397426"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 19:52:28.152898 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:28.152783 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/635b03b0-5944-4a1c-8986-fb3819397426-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "635b03b0-5944-4a1c-8986-fb3819397426" (UID: "635b03b0-5944-4a1c-8986-fb3819397426"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:52:28.152898 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:28.152822 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/635b03b0-5944-4a1c-8986-fb3819397426-istio-token" (OuterVolumeSpecName: "istio-token") pod "635b03b0-5944-4a1c-8986-fb3819397426" (UID: "635b03b0-5944-4a1c-8986-fb3819397426"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:52:28.251520 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:28.251496 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/635b03b0-5944-4a1c-8986-fb3819397426-workload-socket\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:52:28.251520 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:28.251517 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/635b03b0-5944-4a1c-8986-fb3819397426-istio-token\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:52:28.251661 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:28.251531 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qckdl\" (UniqueName: \"kubernetes.io/projected/635b03b0-5944-4a1c-8986-fb3819397426-kube-api-access-qckdl\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:52:28.251661 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:28.251544 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/635b03b0-5944-4a1c-8986-fb3819397426-istio-podinfo\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:52:28.251661 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:28.251556 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/635b03b0-5944-4a1c-8986-fb3819397426-credential-socket\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:52:28.251661 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:28.251568 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/635b03b0-5944-4a1c-8986-fb3819397426-istio-envoy\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:52:28.251661 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:28.251580 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/635b03b0-5944-4a1c-8986-fb3819397426-istio-data\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:52:28.890487 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:28.890462 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk" Jul 06 19:52:28.890853 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:28.890496 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk" event={"ID":"635b03b0-5944-4a1c-8986-fb3819397426","Type":"ContainerDied","Data":"124928ccf201656c8d9cc9ace250387cab2aacbcf918ee1257fe8b93131373a8"} Jul 06 19:52:28.890853 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:28.890555 2575 scope.go:117] "RemoveContainer" containerID="d5426cfc6b6a5a37d568ec732614ea239553d2eb3b182d885374d321938f0b7a" Jul 06 19:52:28.921308 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:28.920853 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk"] Jul 06 19:52:28.926096 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:28.926072 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-6acba6da-openshift-default-f49577fc9-9xhrk"] Jul 06 19:52:29.795287 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:29.795254 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="635b03b0-5944-4a1c-8986-fb3819397426" path="/var/lib/kubelet/pods/635b03b0-5944-4a1c-8986-fb3819397426/volumes" Jul 06 19:52:30.541932 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:30.541901 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-a583ee7e-956db58ff-dfm5b"] Jul 06 19:52:30.542622 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:30.542606 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="635b03b0-5944-4a1c-8986-fb3819397426" containerName="istio-proxy" Jul 06 19:52:30.542668 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:30.542626 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="635b03b0-5944-4a1c-8986-fb3819397426" containerName="istio-proxy" Jul 06 19:52:30.542772 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:30.542761 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="635b03b0-5944-4a1c-8986-fb3819397426" containerName="istio-proxy" Jul 06 19:52:30.546080 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:30.546059 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-a583ee7e-956db58ff-dfm5b" Jul 06 19:52:30.552842 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:30.552813 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-a583ee7e-serving-cert\"" Jul 06 19:52:30.552968 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:30.552910 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-q8w7d\"" Jul 06 19:52:30.557636 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:30.557611 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-a583ee7e-956db58ff-dfm5b"] Jul 06 19:52:30.673292 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:30.673266 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/225bb55f-8f2b-40de-a4ba-a0943bf50e51-maas-api-tls\") pod \"maas-api-e2e-ait-a583ee7e-956db58ff-dfm5b\" (UID: \"225bb55f-8f2b-40de-a4ba-a0943bf50e51\") " pod="opendatahub/maas-api-e2e-ait-a583ee7e-956db58ff-dfm5b" Jul 06 19:52:30.673420 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:30.673322 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4vvd\" (UniqueName: \"kubernetes.io/projected/225bb55f-8f2b-40de-a4ba-a0943bf50e51-kube-api-access-x4vvd\") pod \"maas-api-e2e-ait-a583ee7e-956db58ff-dfm5b\" (UID: \"225bb55f-8f2b-40de-a4ba-a0943bf50e51\") " pod="opendatahub/maas-api-e2e-ait-a583ee7e-956db58ff-dfm5b" Jul 06 19:52:30.673474 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:30.673456 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/225bb55f-8f2b-40de-a4ba-a0943bf50e51-openshift-service-ca\") pod \"maas-api-e2e-ait-a583ee7e-956db58ff-dfm5b\" (UID: \"225bb55f-8f2b-40de-a4ba-a0943bf50e51\") " pod="opendatahub/maas-api-e2e-ait-a583ee7e-956db58ff-dfm5b" Jul 06 19:52:30.773858 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:30.773835 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/225bb55f-8f2b-40de-a4ba-a0943bf50e51-openshift-service-ca\") pod \"maas-api-e2e-ait-a583ee7e-956db58ff-dfm5b\" (UID: \"225bb55f-8f2b-40de-a4ba-a0943bf50e51\") " pod="opendatahub/maas-api-e2e-ait-a583ee7e-956db58ff-dfm5b" Jul 06 19:52:30.773979 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:30.773878 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/225bb55f-8f2b-40de-a4ba-a0943bf50e51-maas-api-tls\") pod \"maas-api-e2e-ait-a583ee7e-956db58ff-dfm5b\" (UID: \"225bb55f-8f2b-40de-a4ba-a0943bf50e51\") " pod="opendatahub/maas-api-e2e-ait-a583ee7e-956db58ff-dfm5b" Jul 06 19:52:30.773979 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:30.773913 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4vvd\" (UniqueName: \"kubernetes.io/projected/225bb55f-8f2b-40de-a4ba-a0943bf50e51-kube-api-access-x4vvd\") pod \"maas-api-e2e-ait-a583ee7e-956db58ff-dfm5b\" (UID: \"225bb55f-8f2b-40de-a4ba-a0943bf50e51\") " pod="opendatahub/maas-api-e2e-ait-a583ee7e-956db58ff-dfm5b" Jul 06 19:52:30.774091 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:52:30.774015 2575 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-a583ee7e-serving-cert: secret "maas-api-e2e-ait-a583ee7e-serving-cert" not found Jul 06 19:52:30.774143 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:52:30.774111 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/225bb55f-8f2b-40de-a4ba-a0943bf50e51-maas-api-tls podName:225bb55f-8f2b-40de-a4ba-a0943bf50e51 nodeName:}" failed. No retries permitted until 2026-07-06 19:52:31.274080244 +0000 UTC m=+1560.094168949 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/225bb55f-8f2b-40de-a4ba-a0943bf50e51-maas-api-tls") pod "maas-api-e2e-ait-a583ee7e-956db58ff-dfm5b" (UID: "225bb55f-8f2b-40de-a4ba-a0943bf50e51") : secret "maas-api-e2e-ait-a583ee7e-serving-cert" not found Jul 06 19:52:30.774492 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:30.774473 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/225bb55f-8f2b-40de-a4ba-a0943bf50e51-openshift-service-ca\") pod \"maas-api-e2e-ait-a583ee7e-956db58ff-dfm5b\" (UID: \"225bb55f-8f2b-40de-a4ba-a0943bf50e51\") " pod="opendatahub/maas-api-e2e-ait-a583ee7e-956db58ff-dfm5b" Jul 06 19:52:30.788298 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:30.788270 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4vvd\" (UniqueName: \"kubernetes.io/projected/225bb55f-8f2b-40de-a4ba-a0943bf50e51-kube-api-access-x4vvd\") pod \"maas-api-e2e-ait-a583ee7e-956db58ff-dfm5b\" (UID: \"225bb55f-8f2b-40de-a4ba-a0943bf50e51\") " pod="opendatahub/maas-api-e2e-ait-a583ee7e-956db58ff-dfm5b" Jul 06 19:52:31.280262 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:31.280229 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/225bb55f-8f2b-40de-a4ba-a0943bf50e51-maas-api-tls\") pod \"maas-api-e2e-ait-a583ee7e-956db58ff-dfm5b\" (UID: \"225bb55f-8f2b-40de-a4ba-a0943bf50e51\") " pod="opendatahub/maas-api-e2e-ait-a583ee7e-956db58ff-dfm5b" Jul 06 19:52:31.282477 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:31.282459 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/225bb55f-8f2b-40de-a4ba-a0943bf50e51-maas-api-tls\") pod \"maas-api-e2e-ait-a583ee7e-956db58ff-dfm5b\" (UID: \"225bb55f-8f2b-40de-a4ba-a0943bf50e51\") " pod="opendatahub/maas-api-e2e-ait-a583ee7e-956db58ff-dfm5b" Jul 06 19:52:31.458861 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:31.458832 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-a583ee7e-956db58ff-dfm5b" Jul 06 19:52:31.600428 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:31.600398 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-a583ee7e-956db58ff-dfm5b"] Jul 06 19:52:31.600813 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:52:31.600600 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod225bb55f_8f2b_40de_a4ba_a0943bf50e51.slice/crio-da2508f9e2f5de73caa17d02fe86ae8d23b24725bcbde5d7572f95e394627a04 WatchSource:0}: Error finding container da2508f9e2f5de73caa17d02fe86ae8d23b24725bcbde5d7572f95e394627a04: Status 404 returned error can't find the container with id da2508f9e2f5de73caa17d02fe86ae8d23b24725bcbde5d7572f95e394627a04 Jul 06 19:52:31.908700 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:31.908665 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-a583ee7e-956db58ff-dfm5b" event={"ID":"225bb55f-8f2b-40de-a4ba-a0943bf50e51","Type":"ContainerStarted","Data":"da2508f9e2f5de73caa17d02fe86ae8d23b24725bcbde5d7572f95e394627a04"} Jul 06 19:52:33.926648 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:33.926618 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-a583ee7e-956db58ff-dfm5b" event={"ID":"225bb55f-8f2b-40de-a4ba-a0943bf50e51","Type":"ContainerStarted","Data":"91c603719f4a7f9eb9b017669a38e68fa9a013912e6d3497d4d3d5b8daaf04fc"} Jul 06 19:52:33.927019 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:33.926747 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-a583ee7e-956db58ff-dfm5b" Jul 06 19:52:33.945235 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:33.945066 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-a583ee7e-956db58ff-dfm5b" podStartSLOduration=2.334919959 podStartE2EDuration="3.945047628s" totalImagesPullingTime="1.610127669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:52:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:52:33.944237435 +0000 UTC m=+1562.764326145" watchObservedRunningTime="2026-07-06 19:52:33.945047628 +0000 UTC m=+1562.765136338" Jul 06 19:52:35.933132 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:35.933093 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-a583ee7e-956db58ff-dfm5b"] Jul 06 19:52:35.938341 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:35.938308 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-a583ee7e-956db58ff-dfm5b" podUID="225bb55f-8f2b-40de-a4ba-a0943bf50e51" containerName="maas-api" containerID="cri-o://91c603719f4a7f9eb9b017669a38e68fa9a013912e6d3497d4d3d5b8daaf04fc" gracePeriod=30 Jul 06 19:52:36.200568 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:36.200544 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-a583ee7e-956db58ff-dfm5b" Jul 06 19:52:36.237247 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:36.237224 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/225bb55f-8f2b-40de-a4ba-a0943bf50e51-maas-api-tls\") pod \"225bb55f-8f2b-40de-a4ba-a0943bf50e51\" (UID: \"225bb55f-8f2b-40de-a4ba-a0943bf50e51\") " Jul 06 19:52:36.237397 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:36.237269 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4vvd\" (UniqueName: \"kubernetes.io/projected/225bb55f-8f2b-40de-a4ba-a0943bf50e51-kube-api-access-x4vvd\") pod \"225bb55f-8f2b-40de-a4ba-a0943bf50e51\" (UID: \"225bb55f-8f2b-40de-a4ba-a0943bf50e51\") " Jul 06 19:52:36.237397 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:36.237321 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/225bb55f-8f2b-40de-a4ba-a0943bf50e51-openshift-service-ca\") pod \"225bb55f-8f2b-40de-a4ba-a0943bf50e51\" (UID: \"225bb55f-8f2b-40de-a4ba-a0943bf50e51\") " Jul 06 19:52:36.237711 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:36.237682 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/225bb55f-8f2b-40de-a4ba-a0943bf50e51-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "225bb55f-8f2b-40de-a4ba-a0943bf50e51" (UID: "225bb55f-8f2b-40de-a4ba-a0943bf50e51"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:52:36.239225 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:36.239199 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/225bb55f-8f2b-40de-a4ba-a0943bf50e51-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "225bb55f-8f2b-40de-a4ba-a0943bf50e51" (UID: "225bb55f-8f2b-40de-a4ba-a0943bf50e51"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:52:36.239567 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:36.239544 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/225bb55f-8f2b-40de-a4ba-a0943bf50e51-kube-api-access-x4vvd" (OuterVolumeSpecName: "kube-api-access-x4vvd") pod "225bb55f-8f2b-40de-a4ba-a0943bf50e51" (UID: "225bb55f-8f2b-40de-a4ba-a0943bf50e51"). InnerVolumeSpecName "kube-api-access-x4vvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:52:36.338545 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:36.338523 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/225bb55f-8f2b-40de-a4ba-a0943bf50e51-maas-api-tls\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:52:36.338545 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:36.338543 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4vvd\" (UniqueName: \"kubernetes.io/projected/225bb55f-8f2b-40de-a4ba-a0943bf50e51-kube-api-access-x4vvd\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:52:36.338661 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:36.338554 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/225bb55f-8f2b-40de-a4ba-a0943bf50e51-openshift-service-ca\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:52:36.945540 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:36.945510 2575 generic.go:356] "Generic (PLEG): container finished" podID="225bb55f-8f2b-40de-a4ba-a0943bf50e51" containerID="91c603719f4a7f9eb9b017669a38e68fa9a013912e6d3497d4d3d5b8daaf04fc" exitCode=0 Jul 06 19:52:36.946021 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:36.945561 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-a583ee7e-956db58ff-dfm5b" Jul 06 19:52:36.946021 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:36.945592 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-a583ee7e-956db58ff-dfm5b" event={"ID":"225bb55f-8f2b-40de-a4ba-a0943bf50e51","Type":"ContainerDied","Data":"91c603719f4a7f9eb9b017669a38e68fa9a013912e6d3497d4d3d5b8daaf04fc"} Jul 06 19:52:36.946021 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:36.945642 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-a583ee7e-956db58ff-dfm5b" event={"ID":"225bb55f-8f2b-40de-a4ba-a0943bf50e51","Type":"ContainerDied","Data":"da2508f9e2f5de73caa17d02fe86ae8d23b24725bcbde5d7572f95e394627a04"} Jul 06 19:52:36.946021 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:36.945664 2575 scope.go:117] "RemoveContainer" containerID="91c603719f4a7f9eb9b017669a38e68fa9a013912e6d3497d4d3d5b8daaf04fc" Jul 06 19:52:36.955504 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:36.955489 2575 scope.go:117] "RemoveContainer" containerID="91c603719f4a7f9eb9b017669a38e68fa9a013912e6d3497d4d3d5b8daaf04fc" Jul 06 19:52:36.955732 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:52:36.955714 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91c603719f4a7f9eb9b017669a38e68fa9a013912e6d3497d4d3d5b8daaf04fc\": container with ID starting with 91c603719f4a7f9eb9b017669a38e68fa9a013912e6d3497d4d3d5b8daaf04fc not found: ID does not exist" containerID="91c603719f4a7f9eb9b017669a38e68fa9a013912e6d3497d4d3d5b8daaf04fc" Jul 06 19:52:36.955792 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:36.955743 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91c603719f4a7f9eb9b017669a38e68fa9a013912e6d3497d4d3d5b8daaf04fc"} err="failed to get container status \"91c603719f4a7f9eb9b017669a38e68fa9a013912e6d3497d4d3d5b8daaf04fc\": rpc error: code = NotFound desc = could not find container \"91c603719f4a7f9eb9b017669a38e68fa9a013912e6d3497d4d3d5b8daaf04fc\": container with ID starting with 91c603719f4a7f9eb9b017669a38e68fa9a013912e6d3497d4d3d5b8daaf04fc not found: ID does not exist" Jul 06 19:52:36.969073 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:36.969047 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-a583ee7e-956db58ff-dfm5b"] Jul 06 19:52:36.972972 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:36.972953 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-a583ee7e-956db58ff-dfm5b"] Jul 06 19:52:37.801528 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:37.801494 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="225bb55f-8f2b-40de-a4ba-a0943bf50e51" path="/var/lib/kubelet/pods/225bb55f-8f2b-40de-a4ba-a0943bf50e51/volumes" Jul 06 19:52:49.088659 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:49.088628 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj"] Jul 06 19:52:49.089345 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:49.089325 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="225bb55f-8f2b-40de-a4ba-a0943bf50e51" containerName="maas-api" Jul 06 19:52:49.089407 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:49.089350 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="225bb55f-8f2b-40de-a4ba-a0943bf50e51" containerName="maas-api" Jul 06 19:52:49.089517 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:49.089505 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="225bb55f-8f2b-40de-a4ba-a0943bf50e51" containerName="maas-api" Jul 06 19:52:49.094194 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:49.094161 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj" Jul 06 19:52:49.096991 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:49.096969 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 06 19:52:49.097104 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:49.097040 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-26edcf53-openshift-default-dockercfg-qshw8\"" Jul 06 19:52:49.107301 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:49.107279 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj"] Jul 06 19:52:49.141781 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:49.141755 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a226bac7-ea8a-45e1-8ed7-8afa278ad719-istio-token\") pod \"e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj\" (UID: \"a226bac7-ea8a-45e1-8ed7-8afa278ad719\") " pod="openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj" Jul 06 19:52:49.141915 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:49.141791 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a226bac7-ea8a-45e1-8ed7-8afa278ad719-istio-podinfo\") pod \"e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj\" (UID: \"a226bac7-ea8a-45e1-8ed7-8afa278ad719\") " pod="openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj" Jul 06 19:52:49.141915 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:49.141823 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a226bac7-ea8a-45e1-8ed7-8afa278ad719-istio-envoy\") pod \"e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj\" (UID: \"a226bac7-ea8a-45e1-8ed7-8afa278ad719\") " pod="openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj" Jul 06 19:52:49.142014 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:49.141933 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a226bac7-ea8a-45e1-8ed7-8afa278ad719-credential-socket\") pod \"e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj\" (UID: \"a226bac7-ea8a-45e1-8ed7-8afa278ad719\") " pod="openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj" Jul 06 19:52:49.142014 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:49.141981 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a226bac7-ea8a-45e1-8ed7-8afa278ad719-workload-certs\") pod \"e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj\" (UID: \"a226bac7-ea8a-45e1-8ed7-8afa278ad719\") " pod="openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj" Jul 06 19:52:49.142090 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:49.142011 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a226bac7-ea8a-45e1-8ed7-8afa278ad719-istiod-ca-cert\") pod \"e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj\" (UID: \"a226bac7-ea8a-45e1-8ed7-8afa278ad719\") " pod="openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj" Jul 06 19:52:49.142140 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:49.142093 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ftvj\" (UniqueName: \"kubernetes.io/projected/a226bac7-ea8a-45e1-8ed7-8afa278ad719-kube-api-access-2ftvj\") pod \"e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj\" (UID: \"a226bac7-ea8a-45e1-8ed7-8afa278ad719\") " pod="openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj" Jul 06 19:52:49.142212 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:49.142169 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a226bac7-ea8a-45e1-8ed7-8afa278ad719-workload-socket\") pod \"e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj\" (UID: \"a226bac7-ea8a-45e1-8ed7-8afa278ad719\") " pod="openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj" Jul 06 19:52:49.142256 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:49.142214 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a226bac7-ea8a-45e1-8ed7-8afa278ad719-istio-data\") pod \"e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj\" (UID: \"a226bac7-ea8a-45e1-8ed7-8afa278ad719\") " pod="openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj" Jul 06 19:52:49.243616 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:49.243589 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a226bac7-ea8a-45e1-8ed7-8afa278ad719-istio-envoy\") pod \"e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj\" (UID: \"a226bac7-ea8a-45e1-8ed7-8afa278ad719\") " pod="openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj" Jul 06 19:52:49.243780 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:49.243626 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a226bac7-ea8a-45e1-8ed7-8afa278ad719-credential-socket\") pod \"e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj\" (UID: \"a226bac7-ea8a-45e1-8ed7-8afa278ad719\") " pod="openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj" Jul 06 19:52:49.243780 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:49.243644 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a226bac7-ea8a-45e1-8ed7-8afa278ad719-workload-certs\") pod \"e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj\" (UID: \"a226bac7-ea8a-45e1-8ed7-8afa278ad719\") " pod="openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj" Jul 06 19:52:49.243780 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:49.243660 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a226bac7-ea8a-45e1-8ed7-8afa278ad719-istiod-ca-cert\") pod \"e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj\" (UID: \"a226bac7-ea8a-45e1-8ed7-8afa278ad719\") " pod="openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj" Jul 06 19:52:49.243942 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:49.243803 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ftvj\" (UniqueName: \"kubernetes.io/projected/a226bac7-ea8a-45e1-8ed7-8afa278ad719-kube-api-access-2ftvj\") pod \"e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj\" (UID: \"a226bac7-ea8a-45e1-8ed7-8afa278ad719\") " pod="openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj" Jul 06 19:52:49.243942 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:49.243877 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a226bac7-ea8a-45e1-8ed7-8afa278ad719-workload-socket\") pod \"e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj\" (UID: \"a226bac7-ea8a-45e1-8ed7-8afa278ad719\") " pod="openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj" Jul 06 19:52:49.243942 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:49.243912 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a226bac7-ea8a-45e1-8ed7-8afa278ad719-istio-data\") pod \"e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj\" (UID: \"a226bac7-ea8a-45e1-8ed7-8afa278ad719\") " pod="openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj" Jul 06 19:52:49.244105 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:49.243948 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a226bac7-ea8a-45e1-8ed7-8afa278ad719-istio-token\") pod \"e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj\" (UID: \"a226bac7-ea8a-45e1-8ed7-8afa278ad719\") " pod="openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj" Jul 06 19:52:49.244105 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:49.243988 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a226bac7-ea8a-45e1-8ed7-8afa278ad719-istio-podinfo\") pod \"e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj\" (UID: \"a226bac7-ea8a-45e1-8ed7-8afa278ad719\") " pod="openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj" Jul 06 19:52:49.244105 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:49.244055 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a226bac7-ea8a-45e1-8ed7-8afa278ad719-credential-socket\") pod \"e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj\" (UID: \"a226bac7-ea8a-45e1-8ed7-8afa278ad719\") " pod="openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj" Jul 06 19:52:49.244408 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:49.244384 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a226bac7-ea8a-45e1-8ed7-8afa278ad719-workload-certs\") pod \"e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj\" (UID: \"a226bac7-ea8a-45e1-8ed7-8afa278ad719\") " pod="openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj" Jul 06 19:52:49.244575 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:49.244544 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a226bac7-ea8a-45e1-8ed7-8afa278ad719-istio-data\") pod \"e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj\" (UID: \"a226bac7-ea8a-45e1-8ed7-8afa278ad719\") " pod="openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj" Jul 06 19:52:49.244714 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:49.244691 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a226bac7-ea8a-45e1-8ed7-8afa278ad719-workload-socket\") pod \"e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj\" (UID: \"a226bac7-ea8a-45e1-8ed7-8afa278ad719\") " pod="openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj" Jul 06 19:52:49.244786 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:49.244749 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a226bac7-ea8a-45e1-8ed7-8afa278ad719-istiod-ca-cert\") pod \"e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj\" (UID: \"a226bac7-ea8a-45e1-8ed7-8afa278ad719\") " pod="openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj" Jul 06 19:52:49.245922 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:49.245900 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a226bac7-ea8a-45e1-8ed7-8afa278ad719-istio-envoy\") pod \"e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj\" (UID: \"a226bac7-ea8a-45e1-8ed7-8afa278ad719\") " pod="openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj" Jul 06 19:52:49.246431 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:49.246407 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a226bac7-ea8a-45e1-8ed7-8afa278ad719-istio-podinfo\") pod \"e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj\" (UID: \"a226bac7-ea8a-45e1-8ed7-8afa278ad719\") " pod="openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj" Jul 06 19:52:49.253886 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:49.253863 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a226bac7-ea8a-45e1-8ed7-8afa278ad719-istio-token\") pod \"e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj\" (UID: \"a226bac7-ea8a-45e1-8ed7-8afa278ad719\") " pod="openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj" Jul 06 19:52:49.254218 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:49.254194 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ftvj\" (UniqueName: \"kubernetes.io/projected/a226bac7-ea8a-45e1-8ed7-8afa278ad719-kube-api-access-2ftvj\") pod \"e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj\" (UID: \"a226bac7-ea8a-45e1-8ed7-8afa278ad719\") " pod="openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj" Jul 06 19:52:49.415571 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:49.415486 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj" Jul 06 19:52:49.553351 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:49.553328 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj"] Jul 06 19:52:49.555725 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:52:49.555697 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda226bac7_ea8a_45e1_8ed7_8afa278ad719.slice/crio-c72474fb0b83d43154b42c0ca2ca14f5c9192ee7d6c1a94535a8b6805c02f4d8 WatchSource:0}: Error finding container c72474fb0b83d43154b42c0ca2ca14f5c9192ee7d6c1a94535a8b6805c02f4d8: Status 404 returned error can't find the container with id c72474fb0b83d43154b42c0ca2ca14f5c9192ee7d6c1a94535a8b6805c02f4d8 Jul 06 19:52:49.557712 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:49.557678 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 19:52:49.557793 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:49.557743 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 19:52:49.557793 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:49.557772 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 19:52:50.030040 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:50.030007 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj" event={"ID":"a226bac7-ea8a-45e1-8ed7-8afa278ad719","Type":"ContainerStarted","Data":"6cb4974b76bdc08aafced4aa7560490379c07d07eeb59586b1611033296ad5a0"} Jul 06 19:52:50.030220 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:50.030061 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj" event={"ID":"a226bac7-ea8a-45e1-8ed7-8afa278ad719","Type":"ContainerStarted","Data":"c72474fb0b83d43154b42c0ca2ca14f5c9192ee7d6c1a94535a8b6805c02f4d8"} Jul 06 19:52:50.069434 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:50.069392 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj" podStartSLOduration=1.069379484 podStartE2EDuration="1.069379484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:52:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:52:50.066941231 +0000 UTC m=+1578.887029941" watchObservedRunningTime="2026-07-06 19:52:50.069379484 +0000 UTC m=+1578.889468194" Jul 06 19:52:50.416495 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:50.416462 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj" Jul 06 19:52:50.420931 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:50.420907 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj" Jul 06 19:52:51.037087 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:51.037032 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj" Jul 06 19:52:51.037802 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:51.037784 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj" Jul 06 19:52:54.841309 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:54.841280 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj"] Jul 06 19:52:54.841684 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:52:54.841451 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj" podUID="a226bac7-ea8a-45e1-8ed7-8afa278ad719" containerName="istio-proxy" containerID="cri-o://6cb4974b76bdc08aafced4aa7560490379c07d07eeb59586b1611033296ad5a0" gracePeriod=30 Jul 06 19:53:00.699137 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:53:00.699112 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj" Jul 06 19:53:00.754619 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:53:00.754566 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a226bac7-ea8a-45e1-8ed7-8afa278ad719-istio-envoy\") pod \"a226bac7-ea8a-45e1-8ed7-8afa278ad719\" (UID: \"a226bac7-ea8a-45e1-8ed7-8afa278ad719\") " Jul 06 19:53:00.754619 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:53:00.754604 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a226bac7-ea8a-45e1-8ed7-8afa278ad719-workload-certs\") pod \"a226bac7-ea8a-45e1-8ed7-8afa278ad719\" (UID: \"a226bac7-ea8a-45e1-8ed7-8afa278ad719\") " Jul 06 19:53:00.754835 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:53:00.754637 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a226bac7-ea8a-45e1-8ed7-8afa278ad719-istio-podinfo\") pod \"a226bac7-ea8a-45e1-8ed7-8afa278ad719\" (UID: \"a226bac7-ea8a-45e1-8ed7-8afa278ad719\") " Jul 06 19:53:00.754835 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:53:00.754655 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a226bac7-ea8a-45e1-8ed7-8afa278ad719-workload-socket\") pod \"a226bac7-ea8a-45e1-8ed7-8afa278ad719\" (UID: \"a226bac7-ea8a-45e1-8ed7-8afa278ad719\") " Jul 06 19:53:00.754835 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:53:00.754694 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ftvj\" (UniqueName: \"kubernetes.io/projected/a226bac7-ea8a-45e1-8ed7-8afa278ad719-kube-api-access-2ftvj\") pod \"a226bac7-ea8a-45e1-8ed7-8afa278ad719\" (UID: \"a226bac7-ea8a-45e1-8ed7-8afa278ad719\") " Jul 06 19:53:00.754835 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:53:00.754708 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a226bac7-ea8a-45e1-8ed7-8afa278ad719-istio-data\") pod \"a226bac7-ea8a-45e1-8ed7-8afa278ad719\" (UID: \"a226bac7-ea8a-45e1-8ed7-8afa278ad719\") " Jul 06 19:53:00.754835 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:53:00.754724 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a226bac7-ea8a-45e1-8ed7-8afa278ad719-istiod-ca-cert\") pod \"a226bac7-ea8a-45e1-8ed7-8afa278ad719\" (UID: \"a226bac7-ea8a-45e1-8ed7-8afa278ad719\") " Jul 06 19:53:00.754835 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:53:00.754816 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a226bac7-ea8a-45e1-8ed7-8afa278ad719-istio-token\") pod \"a226bac7-ea8a-45e1-8ed7-8afa278ad719\" (UID: \"a226bac7-ea8a-45e1-8ed7-8afa278ad719\") " Jul 06 19:53:00.755151 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:53:00.754851 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a226bac7-ea8a-45e1-8ed7-8afa278ad719-credential-socket\") pod \"a226bac7-ea8a-45e1-8ed7-8afa278ad719\" (UID: \"a226bac7-ea8a-45e1-8ed7-8afa278ad719\") " Jul 06 19:53:00.755151 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:53:00.754846 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a226bac7-ea8a-45e1-8ed7-8afa278ad719-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a226bac7-ea8a-45e1-8ed7-8afa278ad719" (UID: "a226bac7-ea8a-45e1-8ed7-8afa278ad719"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:53:00.755151 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:53:00.754939 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a226bac7-ea8a-45e1-8ed7-8afa278ad719-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a226bac7-ea8a-45e1-8ed7-8afa278ad719" (UID: "a226bac7-ea8a-45e1-8ed7-8afa278ad719"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:53:00.755373 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:53:00.755226 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a226bac7-ea8a-45e1-8ed7-8afa278ad719-workload-certs\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:53:00.755373 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:53:00.755248 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a226bac7-ea8a-45e1-8ed7-8afa278ad719-workload-socket\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:53:00.755373 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:53:00.755308 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a226bac7-ea8a-45e1-8ed7-8afa278ad719-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a226bac7-ea8a-45e1-8ed7-8afa278ad719" (UID: "a226bac7-ea8a-45e1-8ed7-8afa278ad719"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:53:00.755526 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:53:00.755485 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a226bac7-ea8a-45e1-8ed7-8afa278ad719-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a226bac7-ea8a-45e1-8ed7-8afa278ad719" (UID: "a226bac7-ea8a-45e1-8ed7-8afa278ad719"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:53:00.755718 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:53:00.755697 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a226bac7-ea8a-45e1-8ed7-8afa278ad719-istio-data" (OuterVolumeSpecName: "istio-data") pod "a226bac7-ea8a-45e1-8ed7-8afa278ad719" (UID: "a226bac7-ea8a-45e1-8ed7-8afa278ad719"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:53:00.756776 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:53:00.756753 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a226bac7-ea8a-45e1-8ed7-8afa278ad719-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a226bac7-ea8a-45e1-8ed7-8afa278ad719" (UID: "a226bac7-ea8a-45e1-8ed7-8afa278ad719"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:53:00.756871 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:53:00.756823 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a226bac7-ea8a-45e1-8ed7-8afa278ad719-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a226bac7-ea8a-45e1-8ed7-8afa278ad719" (UID: "a226bac7-ea8a-45e1-8ed7-8afa278ad719"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 19:53:00.757366 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:53:00.757340 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a226bac7-ea8a-45e1-8ed7-8afa278ad719-istio-token" (OuterVolumeSpecName: "istio-token") pod "a226bac7-ea8a-45e1-8ed7-8afa278ad719" (UID: "a226bac7-ea8a-45e1-8ed7-8afa278ad719"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:53:00.757453 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:53:00.757340 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a226bac7-ea8a-45e1-8ed7-8afa278ad719-kube-api-access-2ftvj" (OuterVolumeSpecName: "kube-api-access-2ftvj") pod "a226bac7-ea8a-45e1-8ed7-8afa278ad719" (UID: "a226bac7-ea8a-45e1-8ed7-8afa278ad719"). InnerVolumeSpecName "kube-api-access-2ftvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:53:00.856130 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:53:00.856106 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a226bac7-ea8a-45e1-8ed7-8afa278ad719-istio-token\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:53:00.856240 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:53:00.856133 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a226bac7-ea8a-45e1-8ed7-8afa278ad719-credential-socket\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:53:00.856240 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:53:00.856145 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a226bac7-ea8a-45e1-8ed7-8afa278ad719-istio-envoy\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:53:00.856240 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:53:00.856157 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a226bac7-ea8a-45e1-8ed7-8afa278ad719-istio-podinfo\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:53:00.856240 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:53:00.856168 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ftvj\" (UniqueName: \"kubernetes.io/projected/a226bac7-ea8a-45e1-8ed7-8afa278ad719-kube-api-access-2ftvj\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:53:00.856240 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:53:00.856201 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a226bac7-ea8a-45e1-8ed7-8afa278ad719-istio-data\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:53:00.856240 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:53:00.856213 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a226bac7-ea8a-45e1-8ed7-8afa278ad719-istiod-ca-cert\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:53:01.108579 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:53:01.108557 2575 generic.go:356] "Generic (PLEG): container finished" podID="a226bac7-ea8a-45e1-8ed7-8afa278ad719" containerID="6cb4974b76bdc08aafced4aa7560490379c07d07eeb59586b1611033296ad5a0" exitCode=0 Jul 06 19:53:01.108686 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:53:01.108597 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj" Jul 06 19:53:01.108686 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:53:01.108646 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj" event={"ID":"a226bac7-ea8a-45e1-8ed7-8afa278ad719","Type":"ContainerDied","Data":"6cb4974b76bdc08aafced4aa7560490379c07d07eeb59586b1611033296ad5a0"} Jul 06 19:53:01.108762 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:53:01.108696 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj" event={"ID":"a226bac7-ea8a-45e1-8ed7-8afa278ad719","Type":"ContainerDied","Data":"c72474fb0b83d43154b42c0ca2ca14f5c9192ee7d6c1a94535a8b6805c02f4d8"} Jul 06 19:53:01.108762 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:53:01.108722 2575 scope.go:117] "RemoveContainer" containerID="6cb4974b76bdc08aafced4aa7560490379c07d07eeb59586b1611033296ad5a0" Jul 06 19:53:01.119897 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:53:01.119571 2575 scope.go:117] "RemoveContainer" containerID="6cb4974b76bdc08aafced4aa7560490379c07d07eeb59586b1611033296ad5a0" Jul 06 19:53:01.124403 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:53:01.124377 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cb4974b76bdc08aafced4aa7560490379c07d07eeb59586b1611033296ad5a0\": container with ID starting with 6cb4974b76bdc08aafced4aa7560490379c07d07eeb59586b1611033296ad5a0 not found: ID does not exist" containerID="6cb4974b76bdc08aafced4aa7560490379c07d07eeb59586b1611033296ad5a0" Jul 06 19:53:01.124502 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:53:01.124413 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cb4974b76bdc08aafced4aa7560490379c07d07eeb59586b1611033296ad5a0"} err="failed to get container status \"6cb4974b76bdc08aafced4aa7560490379c07d07eeb59586b1611033296ad5a0\": rpc error: code = NotFound desc = could not find container \"6cb4974b76bdc08aafced4aa7560490379c07d07eeb59586b1611033296ad5a0\": container with ID starting with 6cb4974b76bdc08aafced4aa7560490379c07d07eeb59586b1611033296ad5a0 not found: ID does not exist" Jul 06 19:53:01.138255 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:53:01.138227 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj"] Jul 06 19:53:01.141005 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:53:01.140986 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-26edcf53-openshift-default-7f4758f9c5-5vfqj"] Jul 06 19:53:01.794705 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:53:01.794676 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a226bac7-ea8a-45e1-8ed7-8afa278ad719" path="/var/lib/kubelet/pods/a226bac7-ea8a-45e1-8ed7-8afa278ad719/volumes" Jul 06 19:56:05.636670 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:05.636640 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l"] Jul 06 19:56:05.637135 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:05.637068 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a226bac7-ea8a-45e1-8ed7-8afa278ad719" containerName="istio-proxy" Jul 06 19:56:05.637135 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:05.637079 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a226bac7-ea8a-45e1-8ed7-8afa278ad719" containerName="istio-proxy" Jul 06 19:56:05.637237 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:05.637167 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a226bac7-ea8a-45e1-8ed7-8afa278ad719" containerName="istio-proxy" Jul 06 19:56:05.640485 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:05.640467 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" Jul 06 19:56:05.644217 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:05.644195 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 06 19:56:05.644354 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:05.644228 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d56cc522-openshift-default-dockercfg-6wnfc\"" Jul 06 19:56:05.657271 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:05.657249 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l"] Jul 06 19:56:05.677339 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:05.677308 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65qds\" (UniqueName: \"kubernetes.io/projected/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-kube-api-access-65qds\") pod \"e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l\" (UID: \"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c\") " pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" Jul 06 19:56:05.677460 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:05.677358 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-credential-socket\") pod \"e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l\" (UID: \"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c\") " pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" Jul 06 19:56:05.677460 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:05.677385 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-istio-podinfo\") pod \"e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l\" (UID: \"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c\") " pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" Jul 06 19:56:05.677460 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:05.677421 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-istiod-ca-cert\") pod \"e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l\" (UID: \"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c\") " pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" Jul 06 19:56:05.677460 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:05.677442 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-istio-data\") pod \"e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l\" (UID: \"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c\") " pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" Jul 06 19:56:05.677663 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:05.677497 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-istio-token\") pod \"e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l\" (UID: \"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c\") " pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" Jul 06 19:56:05.677663 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:05.677532 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-workload-certs\") pod \"e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l\" (UID: \"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c\") " pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" Jul 06 19:56:05.677663 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:05.677559 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-workload-socket\") pod \"e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l\" (UID: \"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c\") " pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" Jul 06 19:56:05.677663 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:05.677572 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-istio-envoy\") pod \"e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l\" (UID: \"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c\") " pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" Jul 06 19:56:05.778479 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:05.778454 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-istio-podinfo\") pod \"e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l\" (UID: \"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c\") " pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" Jul 06 19:56:05.778603 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:05.778486 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-istiod-ca-cert\") pod \"e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l\" (UID: \"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c\") " pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" Jul 06 19:56:05.778603 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:05.778508 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-istio-data\") pod \"e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l\" (UID: \"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c\") " pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" Jul 06 19:56:05.778603 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:05.778551 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-istio-token\") pod \"e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l\" (UID: \"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c\") " pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" Jul 06 19:56:05.778603 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:05.778592 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-workload-certs\") pod \"e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l\" (UID: \"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c\") " pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" Jul 06 19:56:05.778798 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:05.778616 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-workload-socket\") pod \"e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l\" (UID: \"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c\") " pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" Jul 06 19:56:05.778798 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:05.778637 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-istio-envoy\") pod \"e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l\" (UID: \"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c\") " pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" Jul 06 19:56:05.778798 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:05.778687 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65qds\" (UniqueName: \"kubernetes.io/projected/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-kube-api-access-65qds\") pod \"e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l\" (UID: \"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c\") " pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" Jul 06 19:56:05.778798 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:05.778748 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-credential-socket\") pod \"e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l\" (UID: \"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c\") " pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" Jul 06 19:56:05.779120 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:05.779074 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-workload-socket\") pod \"e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l\" (UID: \"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c\") " pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" Jul 06 19:56:05.779218 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:05.779133 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-workload-certs\") pod \"e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l\" (UID: \"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c\") " pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" Jul 06 19:56:05.779218 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:05.779171 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-credential-socket\") pod \"e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l\" (UID: \"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c\") " pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" Jul 06 19:56:05.779316 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:05.779294 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-istiod-ca-cert\") pod \"e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l\" (UID: \"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c\") " pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" Jul 06 19:56:05.779352 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:05.779327 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-istio-data\") pod \"e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l\" (UID: \"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c\") " pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" Jul 06 19:56:05.781010 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:05.780989 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-istio-podinfo\") pod \"e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l\" (UID: \"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c\") " pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" Jul 06 19:56:05.781093 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:05.781065 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-istio-envoy\") pod \"e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l\" (UID: \"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c\") " pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" Jul 06 19:56:05.788516 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:05.788497 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-istio-token\") pod \"e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l\" (UID: \"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c\") " pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" Jul 06 19:56:05.788932 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:05.788911 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65qds\" (UniqueName: \"kubernetes.io/projected/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-kube-api-access-65qds\") pod \"e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l\" (UID: \"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c\") " pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" Jul 06 19:56:05.951292 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:05.951224 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" Jul 06 19:56:06.095626 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:06.095596 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l"] Jul 06 19:56:06.097740 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:56:06.097713 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaf5e259_52ae_4cc5_a0c7_285ed7d5787c.slice/crio-b4d5fa4cc4b3d5a90865ccfa6a4d0f7bea985a94138ba6e9a4e3d2b1c234ee26 WatchSource:0}: Error finding container b4d5fa4cc4b3d5a90865ccfa6a4d0f7bea985a94138ba6e9a4e3d2b1c234ee26: Status 404 returned error can't find the container with id b4d5fa4cc4b3d5a90865ccfa6a4d0f7bea985a94138ba6e9a4e3d2b1c234ee26 Jul 06 19:56:06.100392 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:06.100360 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 19:56:06.100507 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:06.100420 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 19:56:06.100507 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:06.100449 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 19:56:06.273692 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:06.273535 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" event={"ID":"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c","Type":"ContainerStarted","Data":"2be927f70c1d5e2533c4b15a270bec7139b59ca82c7bde1f24230ae3ca2e9b01"} Jul 06 19:56:06.273692 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:06.273596 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" event={"ID":"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c","Type":"ContainerStarted","Data":"b4d5fa4cc4b3d5a90865ccfa6a4d0f7bea985a94138ba6e9a4e3d2b1c234ee26"} Jul 06 19:56:06.294752 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:06.294700 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" podStartSLOduration=1.294687797 podStartE2EDuration="1.294687797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:56:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:56:06.292403812 +0000 UTC m=+1775.112492564" watchObservedRunningTime="2026-07-06 19:56:06.294687797 +0000 UTC m=+1775.114776506" Jul 06 19:56:06.951746 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:06.951717 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" Jul 06 19:56:07.957443 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:07.957414 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" Jul 06 19:56:08.285834 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:08.285756 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" Jul 06 19:56:08.286599 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:08.286578 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" Jul 06 19:56:21.370905 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:21.370827 2575 generic.go:356] "Generic (PLEG): container finished" podID="eaf5e259-52ae-4cc5-a0c7-285ed7d5787c" containerID="2be927f70c1d5e2533c4b15a270bec7139b59ca82c7bde1f24230ae3ca2e9b01" exitCode=0 Jul 06 19:56:21.371481 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:21.370895 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" event={"ID":"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c","Type":"ContainerDied","Data":"2be927f70c1d5e2533c4b15a270bec7139b59ca82c7bde1f24230ae3ca2e9b01"} Jul 06 19:56:21.371558 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:21.371512 2575 scope.go:117] "RemoveContainer" containerID="2be927f70c1d5e2533c4b15a270bec7139b59ca82c7bde1f24230ae3ca2e9b01" Jul 06 19:56:21.373201 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:21.372675 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 19:56:21.373201 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:21.372787 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 19:56:21.373201 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:21.372834 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 19:56:22.383888 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:22.383845 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" event={"ID":"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c","Type":"ContainerStarted","Data":"03887434ee92651c4224aed2d42bf4e13107d2eed16114dba6cbb73ddfb6586f"} Jul 06 19:56:22.952279 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:22.952205 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" Jul 06 19:56:22.953352 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:22.953329 2575 patch_prober.go:28] interesting pod/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Jul 06 19:56:22.953471 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:22.953382 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" podUID="eaf5e259-52ae-4cc5-a0c7-285ed7d5787c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Jul 06 19:56:23.286051 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:23.285955 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" Jul 06 19:56:23.951659 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:23.951628 2575 patch_prober.go:28] interesting pod/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Jul 06 19:56:23.952021 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:23.951688 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" podUID="eaf5e259-52ae-4cc5-a0c7-285ed7d5787c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Jul 06 19:56:24.952522 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:24.952485 2575 patch_prober.go:28] interesting pod/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Jul 06 19:56:24.952893 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:24.952540 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" podUID="eaf5e259-52ae-4cc5-a0c7-285ed7d5787c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Jul 06 19:56:25.955601 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:25.955568 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" Jul 06 19:56:25.956716 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:25.956695 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" Jul 06 19:56:27.973439 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:27.973405 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l"] Jul 06 19:56:27.973830 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:27.973584 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" podUID="eaf5e259-52ae-4cc5-a0c7-285ed7d5787c" containerName="istio-proxy" containerID="cri-o://03887434ee92651c4224aed2d42bf4e13107d2eed16114dba6cbb73ddfb6586f" gracePeriod=30 Jul 06 19:56:33.230151 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:33.230129 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" Jul 06 19:56:33.241245 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:33.241170 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65qds\" (UniqueName: \"kubernetes.io/projected/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-kube-api-access-65qds\") pod \"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c\" (UID: \"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c\") " Jul 06 19:56:33.241245 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:33.241235 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-credential-socket\") pod \"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c\" (UID: \"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c\") " Jul 06 19:56:33.241377 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:33.241258 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-istio-envoy\") pod \"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c\" (UID: \"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c\") " Jul 06 19:56:33.241377 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:33.241301 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-istio-data\") pod \"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c\" (UID: \"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c\") " Jul 06 19:56:33.241377 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:33.241335 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-istio-token\") pod \"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c\" (UID: \"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c\") " Jul 06 19:56:33.241377 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:33.241368 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-istio-podinfo\") pod \"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c\" (UID: \"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c\") " Jul 06 19:56:33.241601 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:33.241389 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-istiod-ca-cert\") pod \"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c\" (UID: \"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c\") " Jul 06 19:56:33.241601 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:33.241428 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-workload-certs\") pod \"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c\" (UID: \"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c\") " Jul 06 19:56:33.241601 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:33.241523 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-workload-socket\") pod \"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c\" (UID: \"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c\") " Jul 06 19:56:33.241854 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:33.241517 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "eaf5e259-52ae-4cc5-a0c7-285ed7d5787c" (UID: "eaf5e259-52ae-4cc5-a0c7-285ed7d5787c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:56:33.241916 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:33.241749 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "eaf5e259-52ae-4cc5-a0c7-285ed7d5787c" (UID: "eaf5e259-52ae-4cc5-a0c7-285ed7d5787c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:56:33.241916 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:33.241806 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "eaf5e259-52ae-4cc5-a0c7-285ed7d5787c" (UID: "eaf5e259-52ae-4cc5-a0c7-285ed7d5787c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:56:33.242012 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:33.241929 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "eaf5e259-52ae-4cc5-a0c7-285ed7d5787c" (UID: "eaf5e259-52ae-4cc5-a0c7-285ed7d5787c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:56:33.242230 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:33.242170 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-istio-data" (OuterVolumeSpecName: "istio-data") pod "eaf5e259-52ae-4cc5-a0c7-285ed7d5787c" (UID: "eaf5e259-52ae-4cc5-a0c7-285ed7d5787c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:56:33.243477 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:33.243448 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-istio-token" (OuterVolumeSpecName: "istio-token") pod "eaf5e259-52ae-4cc5-a0c7-285ed7d5787c" (UID: "eaf5e259-52ae-4cc5-a0c7-285ed7d5787c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:56:33.243587 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:33.243563 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "eaf5e259-52ae-4cc5-a0c7-285ed7d5787c" (UID: "eaf5e259-52ae-4cc5-a0c7-285ed7d5787c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:56:33.243651 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:33.243581 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-kube-api-access-65qds" (OuterVolumeSpecName: "kube-api-access-65qds") pod "eaf5e259-52ae-4cc5-a0c7-285ed7d5787c" (UID: "eaf5e259-52ae-4cc5-a0c7-285ed7d5787c"). InnerVolumeSpecName "kube-api-access-65qds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:56:33.243651 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:33.243600 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "eaf5e259-52ae-4cc5-a0c7-285ed7d5787c" (UID: "eaf5e259-52ae-4cc5-a0c7-285ed7d5787c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 19:56:33.342261 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:33.342240 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-istio-podinfo\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:56:33.342261 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:33.342260 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-istiod-ca-cert\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:56:33.342393 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:33.342268 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-workload-certs\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:56:33.342393 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:33.342275 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-workload-socket\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:56:33.342393 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:33.342284 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65qds\" (UniqueName: \"kubernetes.io/projected/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-kube-api-access-65qds\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:56:33.342393 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:33.342294 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-credential-socket\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:56:33.342393 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:33.342304 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-istio-envoy\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:56:33.342393 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:33.342312 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-istio-data\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:56:33.342393 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:33.342319 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c-istio-token\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:56:33.455469 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:33.455442 2575 generic.go:356] "Generic (PLEG): container finished" podID="eaf5e259-52ae-4cc5-a0c7-285ed7d5787c" containerID="03887434ee92651c4224aed2d42bf4e13107d2eed16114dba6cbb73ddfb6586f" exitCode=0 Jul 06 19:56:33.455570 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:33.455488 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" Jul 06 19:56:33.455570 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:33.455528 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" event={"ID":"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c","Type":"ContainerDied","Data":"03887434ee92651c4224aed2d42bf4e13107d2eed16114dba6cbb73ddfb6586f"} Jul 06 19:56:33.455645 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:33.455578 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l" event={"ID":"eaf5e259-52ae-4cc5-a0c7-285ed7d5787c","Type":"ContainerDied","Data":"b4d5fa4cc4b3d5a90865ccfa6a4d0f7bea985a94138ba6e9a4e3d2b1c234ee26"} Jul 06 19:56:33.455645 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:33.455607 2575 scope.go:117] "RemoveContainer" containerID="03887434ee92651c4224aed2d42bf4e13107d2eed16114dba6cbb73ddfb6586f" Jul 06 19:56:33.465578 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:33.465539 2575 scope.go:117] "RemoveContainer" containerID="2be927f70c1d5e2533c4b15a270bec7139b59ca82c7bde1f24230ae3ca2e9b01" Jul 06 19:56:33.474068 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:33.474053 2575 scope.go:117] "RemoveContainer" containerID="03887434ee92651c4224aed2d42bf4e13107d2eed16114dba6cbb73ddfb6586f" Jul 06 19:56:33.474327 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:56:33.474304 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03887434ee92651c4224aed2d42bf4e13107d2eed16114dba6cbb73ddfb6586f\": container with ID starting with 03887434ee92651c4224aed2d42bf4e13107d2eed16114dba6cbb73ddfb6586f not found: ID does not exist" containerID="03887434ee92651c4224aed2d42bf4e13107d2eed16114dba6cbb73ddfb6586f" Jul 06 19:56:33.474394 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:33.474336 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03887434ee92651c4224aed2d42bf4e13107d2eed16114dba6cbb73ddfb6586f"} err="failed to get container status \"03887434ee92651c4224aed2d42bf4e13107d2eed16114dba6cbb73ddfb6586f\": rpc error: code = NotFound desc = could not find container \"03887434ee92651c4224aed2d42bf4e13107d2eed16114dba6cbb73ddfb6586f\": container with ID starting with 03887434ee92651c4224aed2d42bf4e13107d2eed16114dba6cbb73ddfb6586f not found: ID does not exist" Jul 06 19:56:33.474394 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:33.474359 2575 scope.go:117] "RemoveContainer" containerID="2be927f70c1d5e2533c4b15a270bec7139b59ca82c7bde1f24230ae3ca2e9b01" Jul 06 19:56:33.474584 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:56:33.474568 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2be927f70c1d5e2533c4b15a270bec7139b59ca82c7bde1f24230ae3ca2e9b01\": container with ID starting with 2be927f70c1d5e2533c4b15a270bec7139b59ca82c7bde1f24230ae3ca2e9b01 not found: ID does not exist" containerID="2be927f70c1d5e2533c4b15a270bec7139b59ca82c7bde1f24230ae3ca2e9b01" Jul 06 19:56:33.474624 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:33.474597 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2be927f70c1d5e2533c4b15a270bec7139b59ca82c7bde1f24230ae3ca2e9b01"} err="failed to get container status \"2be927f70c1d5e2533c4b15a270bec7139b59ca82c7bde1f24230ae3ca2e9b01\": rpc error: code = NotFound desc = could not find container \"2be927f70c1d5e2533c4b15a270bec7139b59ca82c7bde1f24230ae3ca2e9b01\": container with ID starting with 2be927f70c1d5e2533c4b15a270bec7139b59ca82c7bde1f24230ae3ca2e9b01 not found: ID does not exist" Jul 06 19:56:33.480191 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:33.480158 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l"] Jul 06 19:56:33.486520 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:33.486500 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d56cc522-openshift-default-59d5ff9ff5-jb56l"] Jul 06 19:56:33.795350 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:33.795327 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eaf5e259-52ae-4cc5-a0c7-285ed7d5787c" path="/var/lib/kubelet/pods/eaf5e259-52ae-4cc5-a0c7-285ed7d5787c/volumes" Jul 06 19:56:51.516616 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:51.516581 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-9c52a789-df6cd49df-2cq2z"] Jul 06 19:56:51.517096 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:51.517016 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaf5e259-52ae-4cc5-a0c7-285ed7d5787c" containerName="istio-proxy" Jul 06 19:56:51.517096 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:51.517026 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaf5e259-52ae-4cc5-a0c7-285ed7d5787c" containerName="istio-proxy" Jul 06 19:56:51.517096 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:51.517048 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaf5e259-52ae-4cc5-a0c7-285ed7d5787c" containerName="istio-proxy" Jul 06 19:56:51.517096 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:51.517053 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaf5e259-52ae-4cc5-a0c7-285ed7d5787c" containerName="istio-proxy" Jul 06 19:56:51.517364 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:51.517120 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="eaf5e259-52ae-4cc5-a0c7-285ed7d5787c" containerName="istio-proxy" Jul 06 19:56:51.521704 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:51.521683 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-9c52a789-df6cd49df-2cq2z" Jul 06 19:56:51.524142 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:51.524120 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-9c52a789-serving-cert\"" Jul 06 19:56:51.524433 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:51.524417 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-q8w7d\"" Jul 06 19:56:51.529750 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:51.529713 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-9c52a789-df6cd49df-2cq2z"] Jul 06 19:56:51.597764 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:51.597735 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d3e31b6-be04-4387-9503-8f330dd78e76-openshift-service-ca\") pod \"maas-api-e2e-mt-9c52a789-df6cd49df-2cq2z\" (UID: \"4d3e31b6-be04-4387-9503-8f330dd78e76\") " pod="opendatahub/maas-api-e2e-mt-9c52a789-df6cd49df-2cq2z" Jul 06 19:56:51.597918 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:51.597800 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h7r9\" (UniqueName: \"kubernetes.io/projected/4d3e31b6-be04-4387-9503-8f330dd78e76-kube-api-access-2h7r9\") pod \"maas-api-e2e-mt-9c52a789-df6cd49df-2cq2z\" (UID: \"4d3e31b6-be04-4387-9503-8f330dd78e76\") " pod="opendatahub/maas-api-e2e-mt-9c52a789-df6cd49df-2cq2z" Jul 06 19:56:51.597918 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:51.597853 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d3e31b6-be04-4387-9503-8f330dd78e76-maas-api-tls\") pod \"maas-api-e2e-mt-9c52a789-df6cd49df-2cq2z\" (UID: \"4d3e31b6-be04-4387-9503-8f330dd78e76\") " pod="opendatahub/maas-api-e2e-mt-9c52a789-df6cd49df-2cq2z" Jul 06 19:56:51.699080 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:51.699050 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h7r9\" (UniqueName: \"kubernetes.io/projected/4d3e31b6-be04-4387-9503-8f330dd78e76-kube-api-access-2h7r9\") pod \"maas-api-e2e-mt-9c52a789-df6cd49df-2cq2z\" (UID: \"4d3e31b6-be04-4387-9503-8f330dd78e76\") " pod="opendatahub/maas-api-e2e-mt-9c52a789-df6cd49df-2cq2z" Jul 06 19:56:51.699269 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:51.699102 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d3e31b6-be04-4387-9503-8f330dd78e76-maas-api-tls\") pod \"maas-api-e2e-mt-9c52a789-df6cd49df-2cq2z\" (UID: \"4d3e31b6-be04-4387-9503-8f330dd78e76\") " pod="opendatahub/maas-api-e2e-mt-9c52a789-df6cd49df-2cq2z" Jul 06 19:56:51.699269 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:51.699133 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d3e31b6-be04-4387-9503-8f330dd78e76-openshift-service-ca\") pod \"maas-api-e2e-mt-9c52a789-df6cd49df-2cq2z\" (UID: \"4d3e31b6-be04-4387-9503-8f330dd78e76\") " pod="opendatahub/maas-api-e2e-mt-9c52a789-df6cd49df-2cq2z" Jul 06 19:56:51.699385 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:56:51.699278 2575 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-9c52a789-serving-cert: secret "maas-api-e2e-mt-9c52a789-serving-cert" not found Jul 06 19:56:51.699385 ip-10-0-135-23 kubenswrapper[2575]: E0706 19:56:51.699342 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d3e31b6-be04-4387-9503-8f330dd78e76-maas-api-tls podName:4d3e31b6-be04-4387-9503-8f330dd78e76 nodeName:}" failed. No retries permitted until 2026-07-06 19:56:52.199327076 +0000 UTC m=+1821.019415765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4d3e31b6-be04-4387-9503-8f330dd78e76-maas-api-tls") pod "maas-api-e2e-mt-9c52a789-df6cd49df-2cq2z" (UID: "4d3e31b6-be04-4387-9503-8f330dd78e76") : secret "maas-api-e2e-mt-9c52a789-serving-cert" not found Jul 06 19:56:51.699724 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:51.699706 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d3e31b6-be04-4387-9503-8f330dd78e76-openshift-service-ca\") pod \"maas-api-e2e-mt-9c52a789-df6cd49df-2cq2z\" (UID: \"4d3e31b6-be04-4387-9503-8f330dd78e76\") " pod="opendatahub/maas-api-e2e-mt-9c52a789-df6cd49df-2cq2z" Jul 06 19:56:51.708500 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:51.708482 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h7r9\" (UniqueName: \"kubernetes.io/projected/4d3e31b6-be04-4387-9503-8f330dd78e76-kube-api-access-2h7r9\") pod \"maas-api-e2e-mt-9c52a789-df6cd49df-2cq2z\" (UID: \"4d3e31b6-be04-4387-9503-8f330dd78e76\") " pod="opendatahub/maas-api-e2e-mt-9c52a789-df6cd49df-2cq2z" Jul 06 19:56:52.203388 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:52.203357 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d3e31b6-be04-4387-9503-8f330dd78e76-maas-api-tls\") pod \"maas-api-e2e-mt-9c52a789-df6cd49df-2cq2z\" (UID: \"4d3e31b6-be04-4387-9503-8f330dd78e76\") " pod="opendatahub/maas-api-e2e-mt-9c52a789-df6cd49df-2cq2z" Jul 06 19:56:52.205934 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:52.205907 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d3e31b6-be04-4387-9503-8f330dd78e76-maas-api-tls\") pod \"maas-api-e2e-mt-9c52a789-df6cd49df-2cq2z\" (UID: \"4d3e31b6-be04-4387-9503-8f330dd78e76\") " pod="opendatahub/maas-api-e2e-mt-9c52a789-df6cd49df-2cq2z" Jul 06 19:56:52.435409 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:52.435382 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-9c52a789-df6cd49df-2cq2z" Jul 06 19:56:52.577336 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:52.577308 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-9c52a789-df6cd49df-2cq2z"] Jul 06 19:56:52.577983 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:56:52.577958 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d3e31b6_be04_4387_9503_8f330dd78e76.slice/crio-c9c057d532a9c9b198f77988ff91079b282a66bb18827a71adf9609488d989ea WatchSource:0}: Error finding container c9c057d532a9c9b198f77988ff91079b282a66bb18827a71adf9609488d989ea: Status 404 returned error can't find the container with id c9c057d532a9c9b198f77988ff91079b282a66bb18827a71adf9609488d989ea Jul 06 19:56:52.586356 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:52.586326 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-9c52a789-df6cd49df-2cq2z" event={"ID":"4d3e31b6-be04-4387-9503-8f330dd78e76","Type":"ContainerStarted","Data":"c9c057d532a9c9b198f77988ff91079b282a66bb18827a71adf9609488d989ea"} Jul 06 19:56:53.594344 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:53.594307 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-9c52a789-df6cd49df-2cq2z" event={"ID":"4d3e31b6-be04-4387-9503-8f330dd78e76","Type":"ContainerStarted","Data":"64f449d4bc083fbabf7dd21eef65cb0d4dcaff5a8dec0506b98246be984a15e9"} Jul 06 19:56:53.594757 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:53.594366 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-9c52a789-df6cd49df-2cq2z" Jul 06 19:56:53.617631 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:53.617587 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-9c52a789-df6cd49df-2cq2z" podStartSLOduration=2.312174886 podStartE2EDuration="2.617572596s" totalImagesPullingTime="305.39771ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:56:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:56:53.614924475 +0000 UTC m=+1822.435013184" watchObservedRunningTime="2026-07-06 19:56:53.617572596 +0000 UTC m=+1822.437661306" Jul 06 19:56:59.606211 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:56:59.606171 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-9c52a789-df6cd49df-2cq2z" Jul 06 19:57:23.637114 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:57:23.636235 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-9c52a789-df6cd49df-2cq2z"] Jul 06 19:57:23.637114 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:57:23.636473 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-9c52a789-df6cd49df-2cq2z" podUID="4d3e31b6-be04-4387-9503-8f330dd78e76" containerName="maas-api" containerID="cri-o://64f449d4bc083fbabf7dd21eef65cb0d4dcaff5a8dec0506b98246be984a15e9" gracePeriod=30 Jul 06 19:57:23.781917 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:57:23.781590 2575 generic.go:356] "Generic (PLEG): container finished" podID="4d3e31b6-be04-4387-9503-8f330dd78e76" containerID="64f449d4bc083fbabf7dd21eef65cb0d4dcaff5a8dec0506b98246be984a15e9" exitCode=0 Jul 06 19:57:23.781917 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:57:23.781709 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-9c52a789-df6cd49df-2cq2z" event={"ID":"4d3e31b6-be04-4387-9503-8f330dd78e76","Type":"ContainerDied","Data":"64f449d4bc083fbabf7dd21eef65cb0d4dcaff5a8dec0506b98246be984a15e9"} Jul 06 19:57:23.885604 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:57:23.885579 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-9c52a789-df6cd49df-2cq2z" Jul 06 19:57:24.055490 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:57:24.055421 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d3e31b6-be04-4387-9503-8f330dd78e76-maas-api-tls\") pod \"4d3e31b6-be04-4387-9503-8f330dd78e76\" (UID: \"4d3e31b6-be04-4387-9503-8f330dd78e76\") " Jul 06 19:57:24.055490 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:57:24.055459 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d3e31b6-be04-4387-9503-8f330dd78e76-openshift-service-ca\") pod \"4d3e31b6-be04-4387-9503-8f330dd78e76\" (UID: \"4d3e31b6-be04-4387-9503-8f330dd78e76\") " Jul 06 19:57:24.055666 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:57:24.055497 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2h7r9\" (UniqueName: \"kubernetes.io/projected/4d3e31b6-be04-4387-9503-8f330dd78e76-kube-api-access-2h7r9\") pod \"4d3e31b6-be04-4387-9503-8f330dd78e76\" (UID: \"4d3e31b6-be04-4387-9503-8f330dd78e76\") " Jul 06 19:57:24.055871 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:57:24.055849 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d3e31b6-be04-4387-9503-8f330dd78e76-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4d3e31b6-be04-4387-9503-8f330dd78e76" (UID: "4d3e31b6-be04-4387-9503-8f330dd78e76"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:57:24.057357 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:57:24.057335 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d3e31b6-be04-4387-9503-8f330dd78e76-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4d3e31b6-be04-4387-9503-8f330dd78e76" (UID: "4d3e31b6-be04-4387-9503-8f330dd78e76"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:57:24.057460 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:57:24.057420 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d3e31b6-be04-4387-9503-8f330dd78e76-kube-api-access-2h7r9" (OuterVolumeSpecName: "kube-api-access-2h7r9") pod "4d3e31b6-be04-4387-9503-8f330dd78e76" (UID: "4d3e31b6-be04-4387-9503-8f330dd78e76"). InnerVolumeSpecName "kube-api-access-2h7r9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:57:24.156962 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:57:24.156934 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h7r9\" (UniqueName: \"kubernetes.io/projected/4d3e31b6-be04-4387-9503-8f330dd78e76-kube-api-access-2h7r9\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:57:24.156962 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:57:24.156959 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d3e31b6-be04-4387-9503-8f330dd78e76-maas-api-tls\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:57:24.157097 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:57:24.156973 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d3e31b6-be04-4387-9503-8f330dd78e76-openshift-service-ca\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 19:57:24.789465 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:57:24.789438 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-9c52a789-df6cd49df-2cq2z" Jul 06 19:57:24.789848 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:57:24.789436 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-9c52a789-df6cd49df-2cq2z" event={"ID":"4d3e31b6-be04-4387-9503-8f330dd78e76","Type":"ContainerDied","Data":"c9c057d532a9c9b198f77988ff91079b282a66bb18827a71adf9609488d989ea"} Jul 06 19:57:24.789848 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:57:24.789568 2575 scope.go:117] "RemoveContainer" containerID="64f449d4bc083fbabf7dd21eef65cb0d4dcaff5a8dec0506b98246be984a15e9" Jul 06 19:57:24.812319 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:57:24.812295 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-9c52a789-df6cd49df-2cq2z"] Jul 06 19:57:24.816563 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:57:24.816544 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-9c52a789-df6cd49df-2cq2z"] Jul 06 19:57:25.795642 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:57:25.795600 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d3e31b6-be04-4387-9503-8f330dd78e76" path="/var/lib/kubelet/pods/4d3e31b6-be04-4387-9503-8f330dd78e76/volumes" Jul 06 19:59:24.358200 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:24.358120 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl"] Jul 06 19:59:24.358619 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:24.358601 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d3e31b6-be04-4387-9503-8f330dd78e76" containerName="maas-api" Jul 06 19:59:24.358662 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:24.358621 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d3e31b6-be04-4387-9503-8f330dd78e76" containerName="maas-api" Jul 06 19:59:24.358718 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:24.358709 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d3e31b6-be04-4387-9503-8f330dd78e76" containerName="maas-api" Jul 06 19:59:24.358754 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:24.358722 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="eaf5e259-52ae-4cc5-a0c7-285ed7d5787c" containerName="istio-proxy" Jul 06 19:59:24.361003 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:24.360986 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" Jul 06 19:59:24.363938 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:24.363917 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-62de5f-openshift-default-dockercfg-5g6pb\"" Jul 06 19:59:24.364234 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:24.364214 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 06 19:59:24.374071 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:24.374047 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl"] Jul 06 19:59:24.439569 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:24.439548 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30d567b7-e41c-4528-b99f-b2f13dd07845-istiod-ca-cert\") pod \"e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl\" (UID: \"30d567b7-e41c-4528-b99f-b2f13dd07845\") " pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" Jul 06 19:59:24.439662 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:24.439587 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drjtv\" (UniqueName: \"kubernetes.io/projected/30d567b7-e41c-4528-b99f-b2f13dd07845-kube-api-access-drjtv\") pod \"e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl\" (UID: \"30d567b7-e41c-4528-b99f-b2f13dd07845\") " pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" Jul 06 19:59:24.439662 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:24.439635 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d567b7-e41c-4528-b99f-b2f13dd07845-workload-socket\") pod \"e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl\" (UID: \"30d567b7-e41c-4528-b99f-b2f13dd07845\") " pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" Jul 06 19:59:24.439739 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:24.439671 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30d567b7-e41c-4528-b99f-b2f13dd07845-istio-envoy\") pod \"e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl\" (UID: \"30d567b7-e41c-4528-b99f-b2f13dd07845\") " pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" Jul 06 19:59:24.439739 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:24.439731 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30d567b7-e41c-4528-b99f-b2f13dd07845-istio-data\") pod \"e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl\" (UID: \"30d567b7-e41c-4528-b99f-b2f13dd07845\") " pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" Jul 06 19:59:24.439808 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:24.439751 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30d567b7-e41c-4528-b99f-b2f13dd07845-istio-podinfo\") pod \"e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl\" (UID: \"30d567b7-e41c-4528-b99f-b2f13dd07845\") " pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" Jul 06 19:59:24.439808 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:24.439770 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30d567b7-e41c-4528-b99f-b2f13dd07845-istio-token\") pod \"e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl\" (UID: \"30d567b7-e41c-4528-b99f-b2f13dd07845\") " pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" Jul 06 19:59:24.439808 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:24.439790 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d567b7-e41c-4528-b99f-b2f13dd07845-credential-socket\") pod \"e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl\" (UID: \"30d567b7-e41c-4528-b99f-b2f13dd07845\") " pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" Jul 06 19:59:24.439903 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:24.439857 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30d567b7-e41c-4528-b99f-b2f13dd07845-workload-certs\") pod \"e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl\" (UID: \"30d567b7-e41c-4528-b99f-b2f13dd07845\") " pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" Jul 06 19:59:24.540395 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:24.540375 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d567b7-e41c-4528-b99f-b2f13dd07845-workload-socket\") pod \"e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl\" (UID: \"30d567b7-e41c-4528-b99f-b2f13dd07845\") " pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" Jul 06 19:59:24.540488 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:24.540402 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30d567b7-e41c-4528-b99f-b2f13dd07845-istio-envoy\") pod \"e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl\" (UID: \"30d567b7-e41c-4528-b99f-b2f13dd07845\") " pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" Jul 06 19:59:24.540488 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:24.540429 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30d567b7-e41c-4528-b99f-b2f13dd07845-istio-data\") pod \"e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl\" (UID: \"30d567b7-e41c-4528-b99f-b2f13dd07845\") " pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" Jul 06 19:59:24.540488 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:24.540452 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30d567b7-e41c-4528-b99f-b2f13dd07845-istio-podinfo\") pod \"e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl\" (UID: \"30d567b7-e41c-4528-b99f-b2f13dd07845\") " pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" Jul 06 19:59:24.540488 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:24.540479 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30d567b7-e41c-4528-b99f-b2f13dd07845-istio-token\") pod \"e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl\" (UID: \"30d567b7-e41c-4528-b99f-b2f13dd07845\") " pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" Jul 06 19:59:24.540665 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:24.540507 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d567b7-e41c-4528-b99f-b2f13dd07845-credential-socket\") pod \"e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl\" (UID: \"30d567b7-e41c-4528-b99f-b2f13dd07845\") " pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" Jul 06 19:59:24.540665 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:24.540532 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30d567b7-e41c-4528-b99f-b2f13dd07845-workload-certs\") pod \"e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl\" (UID: \"30d567b7-e41c-4528-b99f-b2f13dd07845\") " pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" Jul 06 19:59:24.540665 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:24.540593 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30d567b7-e41c-4528-b99f-b2f13dd07845-istiod-ca-cert\") pod \"e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl\" (UID: \"30d567b7-e41c-4528-b99f-b2f13dd07845\") " pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" Jul 06 19:59:24.540665 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:24.540643 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drjtv\" (UniqueName: \"kubernetes.io/projected/30d567b7-e41c-4528-b99f-b2f13dd07845-kube-api-access-drjtv\") pod \"e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl\" (UID: \"30d567b7-e41c-4528-b99f-b2f13dd07845\") " pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" Jul 06 19:59:24.540854 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:24.540776 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d567b7-e41c-4528-b99f-b2f13dd07845-workload-socket\") pod \"e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl\" (UID: \"30d567b7-e41c-4528-b99f-b2f13dd07845\") " pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" Jul 06 19:59:24.540910 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:24.540891 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30d567b7-e41c-4528-b99f-b2f13dd07845-workload-certs\") pod \"e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl\" (UID: \"30d567b7-e41c-4528-b99f-b2f13dd07845\") " pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" Jul 06 19:59:24.541059 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:24.541036 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30d567b7-e41c-4528-b99f-b2f13dd07845-istio-data\") pod \"e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl\" (UID: \"30d567b7-e41c-4528-b99f-b2f13dd07845\") " pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" Jul 06 19:59:24.541124 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:24.541085 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d567b7-e41c-4528-b99f-b2f13dd07845-credential-socket\") pod \"e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl\" (UID: \"30d567b7-e41c-4528-b99f-b2f13dd07845\") " pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" Jul 06 19:59:24.541378 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:24.541357 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30d567b7-e41c-4528-b99f-b2f13dd07845-istiod-ca-cert\") pod \"e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl\" (UID: \"30d567b7-e41c-4528-b99f-b2f13dd07845\") " pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" Jul 06 19:59:24.542609 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:24.542581 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30d567b7-e41c-4528-b99f-b2f13dd07845-istio-envoy\") pod \"e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl\" (UID: \"30d567b7-e41c-4528-b99f-b2f13dd07845\") " pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" Jul 06 19:59:24.543231 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:24.543214 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30d567b7-e41c-4528-b99f-b2f13dd07845-istio-podinfo\") pod \"e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl\" (UID: \"30d567b7-e41c-4528-b99f-b2f13dd07845\") " pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" Jul 06 19:59:24.551146 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:24.550959 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30d567b7-e41c-4528-b99f-b2f13dd07845-istio-token\") pod \"e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl\" (UID: \"30d567b7-e41c-4528-b99f-b2f13dd07845\") " pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" Jul 06 19:59:24.554725 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:24.554703 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drjtv\" (UniqueName: \"kubernetes.io/projected/30d567b7-e41c-4528-b99f-b2f13dd07845-kube-api-access-drjtv\") pod \"e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl\" (UID: \"30d567b7-e41c-4528-b99f-b2f13dd07845\") " pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" Jul 06 19:59:24.675238 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:24.675157 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" Jul 06 19:59:24.812708 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:24.812682 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl"] Jul 06 19:59:24.813672 ip-10-0-135-23 kubenswrapper[2575]: W0706 19:59:24.813647 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30d567b7_e41c_4528_b99f_b2f13dd07845.slice/crio-649c84c17fbba8c3b4abf1f7a72e0e1011282f3a03a7066c786650a5840d3b6d WatchSource:0}: Error finding container 649c84c17fbba8c3b4abf1f7a72e0e1011282f3a03a7066c786650a5840d3b6d: Status 404 returned error can't find the container with id 649c84c17fbba8c3b4abf1f7a72e0e1011282f3a03a7066c786650a5840d3b6d Jul 06 19:59:24.815786 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:24.815752 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 19:59:24.815889 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:24.815829 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 19:59:24.815889 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:24.815871 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 19:59:25.547349 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:25.547315 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" event={"ID":"30d567b7-e41c-4528-b99f-b2f13dd07845","Type":"ContainerStarted","Data":"ac74112281097be7bbd7106cb3b0520da1c0594271a63805246b883bc8a43130"} Jul 06 19:59:25.547717 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:25.547372 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" event={"ID":"30d567b7-e41c-4528-b99f-b2f13dd07845","Type":"ContainerStarted","Data":"649c84c17fbba8c3b4abf1f7a72e0e1011282f3a03a7066c786650a5840d3b6d"} Jul 06 19:59:25.568973 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:25.568922 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" podStartSLOduration=1.568903717 podStartE2EDuration="1.568903717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:59:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:59:25.568760429 +0000 UTC m=+1974.388849160" watchObservedRunningTime="2026-07-06 19:59:25.568903717 +0000 UTC m=+1974.388992426" Jul 06 19:59:25.675668 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:25.675644 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" Jul 06 19:59:25.684024 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:25.684000 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" Jul 06 19:59:26.553923 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:26.553898 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" Jul 06 19:59:26.554775 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:26.554756 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" Jul 06 19:59:40.651695 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:40.651657 2575 generic.go:356] "Generic (PLEG): container finished" podID="30d567b7-e41c-4528-b99f-b2f13dd07845" containerID="ac74112281097be7bbd7106cb3b0520da1c0594271a63805246b883bc8a43130" exitCode=0 Jul 06 19:59:40.652161 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:40.651723 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" event={"ID":"30d567b7-e41c-4528-b99f-b2f13dd07845","Type":"ContainerDied","Data":"ac74112281097be7bbd7106cb3b0520da1c0594271a63805246b883bc8a43130"} Jul 06 19:59:40.652161 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:40.652074 2575 scope.go:117] "RemoveContainer" containerID="ac74112281097be7bbd7106cb3b0520da1c0594271a63805246b883bc8a43130" Jul 06 19:59:40.653104 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:40.653065 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 19:59:40.653190 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:40.653160 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 19:59:40.653244 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:40.653228 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 19:59:41.554553 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:41.554518 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" Jul 06 19:59:41.659134 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:41.659076 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" event={"ID":"30d567b7-e41c-4528-b99f-b2f13dd07845","Type":"ContainerStarted","Data":"6a196af2bfd39bdb2b9af1ce41923f3f4743e1e9ec2bf1e55200a1e960c23d7d"} Jul 06 19:59:41.675984 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:41.675950 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" Jul 06 19:59:41.677319 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:41.677284 2575 patch_prober.go:28] interesting pod/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Jul 06 19:59:41.677438 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:41.677367 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" podUID="30d567b7-e41c-4528-b99f-b2f13dd07845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Jul 06 19:59:42.676317 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:42.676280 2575 patch_prober.go:28] interesting pod/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Jul 06 19:59:42.676807 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:42.676348 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" podUID="30d567b7-e41c-4528-b99f-b2f13dd07845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Jul 06 19:59:43.676380 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:43.676336 2575 patch_prober.go:28] interesting pod/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Jul 06 19:59:43.676974 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:43.676909 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" podUID="30d567b7-e41c-4528-b99f-b2f13dd07845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Jul 06 19:59:44.676009 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:44.675974 2575 patch_prober.go:28] interesting pod/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Jul 06 19:59:44.677148 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:44.677021 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" podUID="30d567b7-e41c-4528-b99f-b2f13dd07845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Jul 06 19:59:45.679861 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:45.679830 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" Jul 06 19:59:45.680351 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:45.680312 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" Jul 06 19:59:45.681118 ip-10-0-135-23 kubenswrapper[2575]: I0706 19:59:45.681099 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" Jul 06 20:00:00.138013 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:00.137978 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-inf-a-d188f8-29722800-87mxx"] Jul 06 20:00:00.142707 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:00.142687 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29722800-xdn8c"] Jul 06 20:00:00.142823 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:00.142801 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-inf-a-d188f8-29722800-87mxx" Jul 06 20:00:00.145619 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:00.145597 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-q8w7d\"" Jul 06 20:00:00.146285 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:00.146266 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-inf-b-62de5f-29722800-2c5kw"] Jul 06 20:00:00.147324 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:00.147307 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722800-xdn8c" Jul 06 20:00:00.151448 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:00.151426 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-inf-a-d188f8-29722800-87mxx"] Jul 06 20:00:00.151566 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:00.151516 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-inf-b-62de5f-29722800-2c5kw" Jul 06 20:00:00.153087 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:00.153063 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722800-xdn8c"] Jul 06 20:00:00.156805 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:00.156781 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-inf-b-62de5f-29722800-2c5kw"] Jul 06 20:00:00.258672 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:00.258652 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqv7p\" (UniqueName: \"kubernetes.io/projected/0b3299f0-f107-470e-b250-91d355cbc94f-kube-api-access-vqv7p\") pod \"maas-api-key-cleanup-e2e-inf-b-62de5f-29722800-2c5kw\" (UID: \"0b3299f0-f107-470e-b250-91d355cbc94f\") " pod="opendatahub/maas-api-key-cleanup-e2e-inf-b-62de5f-29722800-2c5kw" Jul 06 20:00:00.258786 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:00.258697 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4799s\" (UniqueName: \"kubernetes.io/projected/52006610-e3c3-40bb-993a-5c58e1283f26-kube-api-access-4799s\") pod \"maas-api-key-cleanup-29722800-xdn8c\" (UID: \"52006610-e3c3-40bb-993a-5c58e1283f26\") " pod="opendatahub/maas-api-key-cleanup-29722800-xdn8c" Jul 06 20:00:00.258786 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:00.258719 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phtsd\" (UniqueName: \"kubernetes.io/projected/1055eccf-a62e-4581-97c6-528df9fc1dd0-kube-api-access-phtsd\") pod \"maas-api-key-cleanup-e2e-inf-a-d188f8-29722800-87mxx\" (UID: \"1055eccf-a62e-4581-97c6-528df9fc1dd0\") " pod="opendatahub/maas-api-key-cleanup-e2e-inf-a-d188f8-29722800-87mxx" Jul 06 20:00:00.359610 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:00.359585 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqv7p\" (UniqueName: \"kubernetes.io/projected/0b3299f0-f107-470e-b250-91d355cbc94f-kube-api-access-vqv7p\") pod \"maas-api-key-cleanup-e2e-inf-b-62de5f-29722800-2c5kw\" (UID: \"0b3299f0-f107-470e-b250-91d355cbc94f\") " pod="opendatahub/maas-api-key-cleanup-e2e-inf-b-62de5f-29722800-2c5kw" Jul 06 20:00:00.359704 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:00.359629 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4799s\" (UniqueName: \"kubernetes.io/projected/52006610-e3c3-40bb-993a-5c58e1283f26-kube-api-access-4799s\") pod \"maas-api-key-cleanup-29722800-xdn8c\" (UID: \"52006610-e3c3-40bb-993a-5c58e1283f26\") " pod="opendatahub/maas-api-key-cleanup-29722800-xdn8c" Jul 06 20:00:00.359704 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:00.359650 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phtsd\" (UniqueName: \"kubernetes.io/projected/1055eccf-a62e-4581-97c6-528df9fc1dd0-kube-api-access-phtsd\") pod \"maas-api-key-cleanup-e2e-inf-a-d188f8-29722800-87mxx\" (UID: \"1055eccf-a62e-4581-97c6-528df9fc1dd0\") " pod="opendatahub/maas-api-key-cleanup-e2e-inf-a-d188f8-29722800-87mxx" Jul 06 20:00:00.369736 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:00.369706 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4799s\" (UniqueName: \"kubernetes.io/projected/52006610-e3c3-40bb-993a-5c58e1283f26-kube-api-access-4799s\") pod \"maas-api-key-cleanup-29722800-xdn8c\" (UID: \"52006610-e3c3-40bb-993a-5c58e1283f26\") " pod="opendatahub/maas-api-key-cleanup-29722800-xdn8c" Jul 06 20:00:00.370317 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:00.370294 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqv7p\" (UniqueName: \"kubernetes.io/projected/0b3299f0-f107-470e-b250-91d355cbc94f-kube-api-access-vqv7p\") pod \"maas-api-key-cleanup-e2e-inf-b-62de5f-29722800-2c5kw\" (UID: \"0b3299f0-f107-470e-b250-91d355cbc94f\") " pod="opendatahub/maas-api-key-cleanup-e2e-inf-b-62de5f-29722800-2c5kw" Jul 06 20:00:00.370425 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:00.370363 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phtsd\" (UniqueName: \"kubernetes.io/projected/1055eccf-a62e-4581-97c6-528df9fc1dd0-kube-api-access-phtsd\") pod \"maas-api-key-cleanup-e2e-inf-a-d188f8-29722800-87mxx\" (UID: \"1055eccf-a62e-4581-97c6-528df9fc1dd0\") " pod="opendatahub/maas-api-key-cleanup-e2e-inf-a-d188f8-29722800-87mxx" Jul 06 20:00:00.459745 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:00.459679 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-inf-a-d188f8-29722800-87mxx" Jul 06 20:00:00.468978 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:00.468833 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722800-xdn8c" Jul 06 20:00:00.477064 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:00.476914 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-inf-b-62de5f-29722800-2c5kw" Jul 06 20:00:00.624896 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:00.624866 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-inf-a-d188f8-29722800-87mxx"] Jul 06 20:00:00.625372 ip-10-0-135-23 kubenswrapper[2575]: W0706 20:00:00.625344 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1055eccf_a62e_4581_97c6_528df9fc1dd0.slice/crio-929efb6ac10bf19fbba96fbfcf9c06a79744ba963845fcd4e56bfa0bf580d283 WatchSource:0}: Error finding container 929efb6ac10bf19fbba96fbfcf9c06a79744ba963845fcd4e56bfa0bf580d283: Status 404 returned error can't find the container with id 929efb6ac10bf19fbba96fbfcf9c06a79744ba963845fcd4e56bfa0bf580d283 Jul 06 20:00:00.793570 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:00.793507 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-inf-a-d188f8-29722800-87mxx" event={"ID":"1055eccf-a62e-4581-97c6-528df9fc1dd0","Type":"ContainerStarted","Data":"929efb6ac10bf19fbba96fbfcf9c06a79744ba963845fcd4e56bfa0bf580d283"} Jul 06 20:00:00.859529 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:00.859496 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-inf-b-62de5f-29722800-2c5kw"] Jul 06 20:00:00.860761 ip-10-0-135-23 kubenswrapper[2575]: W0706 20:00:00.860739 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b3299f0_f107_470e_b250_91d355cbc94f.slice/crio-611603ff19874b641321bf276a6cca0e6a6a242eac60a660a5fdf2222772a4d9 WatchSource:0}: Error finding container 611603ff19874b641321bf276a6cca0e6a6a242eac60a660a5fdf2222772a4d9: Status 404 returned error can't find the container with id 611603ff19874b641321bf276a6cca0e6a6a242eac60a660a5fdf2222772a4d9 Jul 06 20:00:00.862859 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:00.862836 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722800-xdn8c"] Jul 06 20:00:00.863306 ip-10-0-135-23 kubenswrapper[2575]: W0706 20:00:00.863219 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52006610_e3c3_40bb_993a_5c58e1283f26.slice/crio-fb643793fda35edfb9933d4d6091d08b5ada3d728f8ba49b7d6f8f1e037b5b51 WatchSource:0}: Error finding container fb643793fda35edfb9933d4d6091d08b5ada3d728f8ba49b7d6f8f1e037b5b51: Status 404 returned error can't find the container with id fb643793fda35edfb9933d4d6091d08b5ada3d728f8ba49b7d6f8f1e037b5b51 Jul 06 20:00:01.801683 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:01.801652 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722800-xdn8c" event={"ID":"52006610-e3c3-40bb-993a-5c58e1283f26","Type":"ContainerStarted","Data":"6efa46413cc1cfc5e69a12713ad06f5e9ef7cb9481d078207c5d95ca01c0d18e"} Jul 06 20:00:01.802012 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:01.801708 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722800-xdn8c" event={"ID":"52006610-e3c3-40bb-993a-5c58e1283f26","Type":"ContainerStarted","Data":"fb643793fda35edfb9933d4d6091d08b5ada3d728f8ba49b7d6f8f1e037b5b51"} Jul 06 20:00:01.803337 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:01.803313 2575 generic.go:356] "Generic (PLEG): container finished" podID="1055eccf-a62e-4581-97c6-528df9fc1dd0" containerID="c6a2e1094ec55d6c8d6e70ffcf0eba801bbee6af32dc1533f193b6b385a5b593" exitCode=0 Jul 06 20:00:01.803434 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:01.803397 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-inf-a-d188f8-29722800-87mxx" event={"ID":"1055eccf-a62e-4581-97c6-528df9fc1dd0","Type":"ContainerDied","Data":"c6a2e1094ec55d6c8d6e70ffcf0eba801bbee6af32dc1533f193b6b385a5b593"} Jul 06 20:00:01.804945 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:01.804921 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-inf-b-62de5f-29722800-2c5kw" event={"ID":"0b3299f0-f107-470e-b250-91d355cbc94f","Type":"ContainerStarted","Data":"1cb852d768262e425edd05c2798df801d052b538093772de3310a6cef576d124"} Jul 06 20:00:01.805030 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:01.804959 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-inf-b-62de5f-29722800-2c5kw" event={"ID":"0b3299f0-f107-470e-b250-91d355cbc94f","Type":"ContainerStarted","Data":"611603ff19874b641321bf276a6cca0e6a6a242eac60a660a5fdf2222772a4d9"} Jul 06 20:00:01.820786 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:01.820747 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29722800-xdn8c" podStartSLOduration=1.820735901 podStartE2EDuration="1.820735901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:00:01.818307114 +0000 UTC m=+2010.638395845" watchObservedRunningTime="2026-07-06 20:00:01.820735901 +0000 UTC m=+2010.640824610" Jul 06 20:00:01.849165 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:01.849125 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-e2e-inf-b-62de5f-29722800-2c5kw" podStartSLOduration=1.849114829 podStartE2EDuration="1.849114829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:00:01.847410242 +0000 UTC m=+2010.667498952" watchObservedRunningTime="2026-07-06 20:00:01.849114829 +0000 UTC m=+2010.669203543" Jul 06 20:00:02.813215 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:02.813169 2575 generic.go:356] "Generic (PLEG): container finished" podID="0b3299f0-f107-470e-b250-91d355cbc94f" containerID="1cb852d768262e425edd05c2798df801d052b538093772de3310a6cef576d124" exitCode=0 Jul 06 20:00:02.813600 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:02.813261 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-inf-b-62de5f-29722800-2c5kw" event={"ID":"0b3299f0-f107-470e-b250-91d355cbc94f","Type":"ContainerDied","Data":"1cb852d768262e425edd05c2798df801d052b538093772de3310a6cef576d124"} Jul 06 20:00:02.814651 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:02.814633 2575 generic.go:356] "Generic (PLEG): container finished" podID="52006610-e3c3-40bb-993a-5c58e1283f26" containerID="6efa46413cc1cfc5e69a12713ad06f5e9ef7cb9481d078207c5d95ca01c0d18e" exitCode=0 Jul 06 20:00:02.814737 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:02.814705 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722800-xdn8c" event={"ID":"52006610-e3c3-40bb-993a-5c58e1283f26","Type":"ContainerDied","Data":"6efa46413cc1cfc5e69a12713ad06f5e9ef7cb9481d078207c5d95ca01c0d18e"} Jul 06 20:00:02.955195 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:02.955162 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-inf-a-d188f8-29722800-87mxx" Jul 06 20:00:03.086645 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:03.086615 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phtsd\" (UniqueName: \"kubernetes.io/projected/1055eccf-a62e-4581-97c6-528df9fc1dd0-kube-api-access-phtsd\") pod \"1055eccf-a62e-4581-97c6-528df9fc1dd0\" (UID: \"1055eccf-a62e-4581-97c6-528df9fc1dd0\") " Jul 06 20:00:03.088556 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:03.088531 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1055eccf-a62e-4581-97c6-528df9fc1dd0-kube-api-access-phtsd" (OuterVolumeSpecName: "kube-api-access-phtsd") pod "1055eccf-a62e-4581-97c6-528df9fc1dd0" (UID: "1055eccf-a62e-4581-97c6-528df9fc1dd0"). InnerVolumeSpecName "kube-api-access-phtsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:00:03.187872 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:03.187847 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phtsd\" (UniqueName: \"kubernetes.io/projected/1055eccf-a62e-4581-97c6-528df9fc1dd0-kube-api-access-phtsd\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 20:00:03.821044 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:03.821019 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-inf-a-d188f8-29722800-87mxx" Jul 06 20:00:03.821355 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:03.821055 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-inf-a-d188f8-29722800-87mxx" event={"ID":"1055eccf-a62e-4581-97c6-528df9fc1dd0","Type":"ContainerDied","Data":"929efb6ac10bf19fbba96fbfcf9c06a79744ba963845fcd4e56bfa0bf580d283"} Jul 06 20:00:03.821355 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:03.821098 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="929efb6ac10bf19fbba96fbfcf9c06a79744ba963845fcd4e56bfa0bf580d283" Jul 06 20:00:03.970939 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:03.970917 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722800-xdn8c" Jul 06 20:00:03.974619 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:03.974598 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-inf-b-62de5f-29722800-2c5kw" Jul 06 20:00:04.095614 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:04.095589 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4799s\" (UniqueName: \"kubernetes.io/projected/52006610-e3c3-40bb-993a-5c58e1283f26-kube-api-access-4799s\") pod \"52006610-e3c3-40bb-993a-5c58e1283f26\" (UID: \"52006610-e3c3-40bb-993a-5c58e1283f26\") " Jul 06 20:00:04.095749 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:04.095694 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqv7p\" (UniqueName: \"kubernetes.io/projected/0b3299f0-f107-470e-b250-91d355cbc94f-kube-api-access-vqv7p\") pod \"0b3299f0-f107-470e-b250-91d355cbc94f\" (UID: \"0b3299f0-f107-470e-b250-91d355cbc94f\") " Jul 06 20:00:04.097549 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:04.097522 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52006610-e3c3-40bb-993a-5c58e1283f26-kube-api-access-4799s" (OuterVolumeSpecName: "kube-api-access-4799s") pod "52006610-e3c3-40bb-993a-5c58e1283f26" (UID: "52006610-e3c3-40bb-993a-5c58e1283f26"). InnerVolumeSpecName "kube-api-access-4799s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:00:04.097656 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:04.097553 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b3299f0-f107-470e-b250-91d355cbc94f-kube-api-access-vqv7p" (OuterVolumeSpecName: "kube-api-access-vqv7p") pod "0b3299f0-f107-470e-b250-91d355cbc94f" (UID: "0b3299f0-f107-470e-b250-91d355cbc94f"). InnerVolumeSpecName "kube-api-access-vqv7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:00:04.196786 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:04.196760 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqv7p\" (UniqueName: \"kubernetes.io/projected/0b3299f0-f107-470e-b250-91d355cbc94f-kube-api-access-vqv7p\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 20:00:04.196786 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:04.196788 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4799s\" (UniqueName: \"kubernetes.io/projected/52006610-e3c3-40bb-993a-5c58e1283f26-kube-api-access-4799s\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 20:00:04.827886 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:04.827858 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-inf-b-62de5f-29722800-2c5kw" Jul 06 20:00:04.828288 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:04.827858 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-inf-b-62de5f-29722800-2c5kw" event={"ID":"0b3299f0-f107-470e-b250-91d355cbc94f","Type":"ContainerDied","Data":"611603ff19874b641321bf276a6cca0e6a6a242eac60a660a5fdf2222772a4d9"} Jul 06 20:00:04.828288 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:04.827977 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="611603ff19874b641321bf276a6cca0e6a6a242eac60a660a5fdf2222772a4d9" Jul 06 20:00:04.829263 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:04.829242 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722800-xdn8c" Jul 06 20:00:04.829387 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:04.829242 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722800-xdn8c" event={"ID":"52006610-e3c3-40bb-993a-5c58e1283f26","Type":"ContainerDied","Data":"fb643793fda35edfb9933d4d6091d08b5ada3d728f8ba49b7d6f8f1e037b5b51"} Jul 06 20:00:04.829387 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:04.829364 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb643793fda35edfb9933d4d6091d08b5ada3d728f8ba49b7d6f8f1e037b5b51" Jul 06 20:00:22.660078 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:22.660037 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-inf-a-d188f8-29722800-87mxx"] Jul 06 20:00:22.662722 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:22.662699 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-inf-a-d188f8-29722800-87mxx"] Jul 06 20:00:23.794729 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:00:23.794694 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1055eccf-a62e-4581-97c6-528df9fc1dd0" path="/var/lib/kubelet/pods/1055eccf-a62e-4581-97c6-528df9fc1dd0/volumes" Jul 06 20:01:06.433928 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:06.433898 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-inf-b-62de5f-29722800-2c5kw"] Jul 06 20:01:06.436471 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:06.436449 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-inf-b-62de5f-29722800-2c5kw"] Jul 06 20:01:07.795295 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:07.795262 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b3299f0-f107-470e-b250-91d355cbc94f" path="/var/lib/kubelet/pods/0b3299f0-f107-470e-b250-91d355cbc94f/volumes" Jul 06 20:01:13.096572 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:13.096541 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl"] Jul 06 20:01:13.096969 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:13.096750 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" podUID="30d567b7-e41c-4528-b99f-b2f13dd07845" containerName="istio-proxy" containerID="cri-o://6a196af2bfd39bdb2b9af1ce41923f3f4743e1e9ec2bf1e55200a1e960c23d7d" gracePeriod=30 Jul 06 20:01:15.682451 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:15.682420 2575 patch_prober.go:28] interesting pod/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 20:01:15.682772 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:15.682476 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" podUID="30d567b7-e41c-4528-b99f-b2f13dd07845" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:01:18.315045 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:18.315017 2575 generic.go:356] "Generic (PLEG): container finished" podID="30d567b7-e41c-4528-b99f-b2f13dd07845" containerID="6a196af2bfd39bdb2b9af1ce41923f3f4743e1e9ec2bf1e55200a1e960c23d7d" exitCode=0 Jul 06 20:01:18.315499 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:18.315056 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" event={"ID":"30d567b7-e41c-4528-b99f-b2f13dd07845","Type":"ContainerDied","Data":"6a196af2bfd39bdb2b9af1ce41923f3f4743e1e9ec2bf1e55200a1e960c23d7d"} Jul 06 20:01:18.315499 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:18.315114 2575 scope.go:117] "RemoveContainer" containerID="ac74112281097be7bbd7106cb3b0520da1c0594271a63805246b883bc8a43130" Jul 06 20:01:18.366788 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:18.366732 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" Jul 06 20:01:18.501118 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:18.501091 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d567b7-e41c-4528-b99f-b2f13dd07845-workload-socket\") pod \"30d567b7-e41c-4528-b99f-b2f13dd07845\" (UID: \"30d567b7-e41c-4528-b99f-b2f13dd07845\") " Jul 06 20:01:18.501287 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:18.501124 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drjtv\" (UniqueName: \"kubernetes.io/projected/30d567b7-e41c-4528-b99f-b2f13dd07845-kube-api-access-drjtv\") pod \"30d567b7-e41c-4528-b99f-b2f13dd07845\" (UID: \"30d567b7-e41c-4528-b99f-b2f13dd07845\") " Jul 06 20:01:18.501287 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:18.501142 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30d567b7-e41c-4528-b99f-b2f13dd07845-istio-podinfo\") pod \"30d567b7-e41c-4528-b99f-b2f13dd07845\" (UID: \"30d567b7-e41c-4528-b99f-b2f13dd07845\") " Jul 06 20:01:18.501416 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:18.501296 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30d567b7-e41c-4528-b99f-b2f13dd07845-istio-envoy\") pod \"30d567b7-e41c-4528-b99f-b2f13dd07845\" (UID: \"30d567b7-e41c-4528-b99f-b2f13dd07845\") " Jul 06 20:01:18.501416 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:18.501346 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d567b7-e41c-4528-b99f-b2f13dd07845-credential-socket\") pod \"30d567b7-e41c-4528-b99f-b2f13dd07845\" (UID: \"30d567b7-e41c-4528-b99f-b2f13dd07845\") " Jul 06 20:01:18.501416 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:18.501371 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30d567b7-e41c-4528-b99f-b2f13dd07845-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "30d567b7-e41c-4528-b99f-b2f13dd07845" (UID: "30d567b7-e41c-4528-b99f-b2f13dd07845"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:01:18.501416 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:18.501379 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30d567b7-e41c-4528-b99f-b2f13dd07845-istiod-ca-cert\") pod \"30d567b7-e41c-4528-b99f-b2f13dd07845\" (UID: \"30d567b7-e41c-4528-b99f-b2f13dd07845\") " Jul 06 20:01:18.501611 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:18.501435 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30d567b7-e41c-4528-b99f-b2f13dd07845-istio-token\") pod \"30d567b7-e41c-4528-b99f-b2f13dd07845\" (UID: \"30d567b7-e41c-4528-b99f-b2f13dd07845\") " Jul 06 20:01:18.501611 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:18.501477 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30d567b7-e41c-4528-b99f-b2f13dd07845-istio-data\") pod \"30d567b7-e41c-4528-b99f-b2f13dd07845\" (UID: \"30d567b7-e41c-4528-b99f-b2f13dd07845\") " Jul 06 20:01:18.501611 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:18.501548 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30d567b7-e41c-4528-b99f-b2f13dd07845-workload-certs\") pod \"30d567b7-e41c-4528-b99f-b2f13dd07845\" (UID: \"30d567b7-e41c-4528-b99f-b2f13dd07845\") " Jul 06 20:01:18.501764 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:18.501663 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30d567b7-e41c-4528-b99f-b2f13dd07845-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "30d567b7-e41c-4528-b99f-b2f13dd07845" (UID: "30d567b7-e41c-4528-b99f-b2f13dd07845"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:01:18.501764 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:18.501752 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30d567b7-e41c-4528-b99f-b2f13dd07845-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "30d567b7-e41c-4528-b99f-b2f13dd07845" (UID: "30d567b7-e41c-4528-b99f-b2f13dd07845"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:01:18.501957 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:18.501925 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30d567b7-e41c-4528-b99f-b2f13dd07845-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "30d567b7-e41c-4528-b99f-b2f13dd07845" (UID: "30d567b7-e41c-4528-b99f-b2f13dd07845"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:01:18.501957 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:18.501940 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d567b7-e41c-4528-b99f-b2f13dd07845-credential-socket\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 20:01:18.501957 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:18.501960 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30d567b7-e41c-4528-b99f-b2f13dd07845-istiod-ca-cert\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 20:01:18.502211 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:18.501973 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d567b7-e41c-4528-b99f-b2f13dd07845-workload-socket\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 20:01:18.502284 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:18.502222 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30d567b7-e41c-4528-b99f-b2f13dd07845-istio-data" (OuterVolumeSpecName: "istio-data") pod "30d567b7-e41c-4528-b99f-b2f13dd07845" (UID: "30d567b7-e41c-4528-b99f-b2f13dd07845"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:01:18.503420 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:18.503399 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30d567b7-e41c-4528-b99f-b2f13dd07845-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "30d567b7-e41c-4528-b99f-b2f13dd07845" (UID: "30d567b7-e41c-4528-b99f-b2f13dd07845"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:01:18.503506 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:18.503439 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30d567b7-e41c-4528-b99f-b2f13dd07845-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "30d567b7-e41c-4528-b99f-b2f13dd07845" (UID: "30d567b7-e41c-4528-b99f-b2f13dd07845"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 20:01:18.503846 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:18.503794 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30d567b7-e41c-4528-b99f-b2f13dd07845-istio-token" (OuterVolumeSpecName: "istio-token") pod "30d567b7-e41c-4528-b99f-b2f13dd07845" (UID: "30d567b7-e41c-4528-b99f-b2f13dd07845"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:01:18.503846 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:18.503828 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30d567b7-e41c-4528-b99f-b2f13dd07845-kube-api-access-drjtv" (OuterVolumeSpecName: "kube-api-access-drjtv") pod "30d567b7-e41c-4528-b99f-b2f13dd07845" (UID: "30d567b7-e41c-4528-b99f-b2f13dd07845"). InnerVolumeSpecName "kube-api-access-drjtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:01:18.602708 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:18.602674 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drjtv\" (UniqueName: \"kubernetes.io/projected/30d567b7-e41c-4528-b99f-b2f13dd07845-kube-api-access-drjtv\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 20:01:18.602708 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:18.602703 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30d567b7-e41c-4528-b99f-b2f13dd07845-istio-podinfo\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 20:01:18.602708 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:18.602712 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30d567b7-e41c-4528-b99f-b2f13dd07845-istio-envoy\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 20:01:18.602889 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:18.602720 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30d567b7-e41c-4528-b99f-b2f13dd07845-istio-token\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 20:01:18.602889 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:18.602727 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30d567b7-e41c-4528-b99f-b2f13dd07845-istio-data\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 20:01:18.602889 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:18.602733 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30d567b7-e41c-4528-b99f-b2f13dd07845-workload-certs\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 20:01:19.323365 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:19.323331 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" Jul 06 20:01:19.323750 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:19.323368 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl" event={"ID":"30d567b7-e41c-4528-b99f-b2f13dd07845","Type":"ContainerDied","Data":"649c84c17fbba8c3b4abf1f7a72e0e1011282f3a03a7066c786650a5840d3b6d"} Jul 06 20:01:19.323750 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:19.323418 2575 scope.go:117] "RemoveContainer" containerID="6a196af2bfd39bdb2b9af1ce41923f3f4743e1e9ec2bf1e55200a1e960c23d7d" Jul 06 20:01:19.359627 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:19.359602 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl"] Jul 06 20:01:19.365920 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:19.365898 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-62de5f-openshift-default-765cb8c4d7-8p7jl"] Jul 06 20:01:19.795227 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:19.795125 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30d567b7-e41c-4528-b99f-b2f13dd07845" path="/var/lib/kubelet/pods/30d567b7-e41c-4528-b99f-b2f13dd07845/volumes" Jul 06 20:01:23.460032 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.459992 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt"] Jul 06 20:01:23.461106 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.461078 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52006610-e3c3-40bb-993a-5c58e1283f26" containerName="cleanup" Jul 06 20:01:23.461282 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.461270 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="52006610-e3c3-40bb-993a-5c58e1283f26" containerName="cleanup" Jul 06 20:01:23.461397 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.461376 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1055eccf-a62e-4581-97c6-528df9fc1dd0" containerName="cleanup" Jul 06 20:01:23.461397 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.461391 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1055eccf-a62e-4581-97c6-528df9fc1dd0" containerName="cleanup" Jul 06 20:01:23.461542 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.461426 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30d567b7-e41c-4528-b99f-b2f13dd07845" containerName="istio-proxy" Jul 06 20:01:23.461542 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.461437 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="30d567b7-e41c-4528-b99f-b2f13dd07845" containerName="istio-proxy" Jul 06 20:01:23.461542 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.461461 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b3299f0-f107-470e-b250-91d355cbc94f" containerName="cleanup" Jul 06 20:01:23.461542 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.461469 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b3299f0-f107-470e-b250-91d355cbc94f" containerName="cleanup" Jul 06 20:01:23.461661 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.461594 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="30d567b7-e41c-4528-b99f-b2f13dd07845" containerName="istio-proxy" Jul 06 20:01:23.461661 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.461605 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b3299f0-f107-470e-b250-91d355cbc94f" containerName="cleanup" Jul 06 20:01:23.461661 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.461617 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="30d567b7-e41c-4528-b99f-b2f13dd07845" containerName="istio-proxy" Jul 06 20:01:23.461661 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.461627 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="52006610-e3c3-40bb-993a-5c58e1283f26" containerName="cleanup" Jul 06 20:01:23.461661 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.461635 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1055eccf-a62e-4581-97c6-528df9fc1dd0" containerName="cleanup" Jul 06 20:01:23.466772 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.466755 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" Jul 06 20:01:23.469425 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.469404 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 06 20:01:23.469770 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.469598 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-e82419-openshift-default-dockercfg-mx87t\"" Jul 06 20:01:23.477858 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.477834 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt"] Jul 06 20:01:23.540030 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.540003 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-istio-envoy\") pod \"e2e-api-b-e82419-openshift-default-786996d479-wgfzt\" (UID: \"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a\") " pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" Jul 06 20:01:23.540120 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.540038 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-workload-certs\") pod \"e2e-api-b-e82419-openshift-default-786996d479-wgfzt\" (UID: \"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a\") " pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" Jul 06 20:01:23.540120 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.540067 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-istiod-ca-cert\") pod \"e2e-api-b-e82419-openshift-default-786996d479-wgfzt\" (UID: \"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a\") " pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" Jul 06 20:01:23.540120 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.540096 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-workload-socket\") pod \"e2e-api-b-e82419-openshift-default-786996d479-wgfzt\" (UID: \"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a\") " pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" Jul 06 20:01:23.540120 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.540115 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-credential-socket\") pod \"e2e-api-b-e82419-openshift-default-786996d479-wgfzt\" (UID: \"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a\") " pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" Jul 06 20:01:23.540302 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.540224 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zrfk\" (UniqueName: \"kubernetes.io/projected/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-kube-api-access-4zrfk\") pod \"e2e-api-b-e82419-openshift-default-786996d479-wgfzt\" (UID: \"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a\") " pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" Jul 06 20:01:23.540302 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.540252 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-istio-podinfo\") pod \"e2e-api-b-e82419-openshift-default-786996d479-wgfzt\" (UID: \"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a\") " pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" Jul 06 20:01:23.540302 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.540281 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-istio-token\") pod \"e2e-api-b-e82419-openshift-default-786996d479-wgfzt\" (UID: \"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a\") " pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" Jul 06 20:01:23.540412 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.540306 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-istio-data\") pod \"e2e-api-b-e82419-openshift-default-786996d479-wgfzt\" (UID: \"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a\") " pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" Jul 06 20:01:23.641459 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.641429 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-istio-data\") pod \"e2e-api-b-e82419-openshift-default-786996d479-wgfzt\" (UID: \"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a\") " pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" Jul 06 20:01:23.641594 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.641498 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-istio-envoy\") pod \"e2e-api-b-e82419-openshift-default-786996d479-wgfzt\" (UID: \"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a\") " pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" Jul 06 20:01:23.641594 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.641527 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-workload-certs\") pod \"e2e-api-b-e82419-openshift-default-786996d479-wgfzt\" (UID: \"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a\") " pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" Jul 06 20:01:23.641594 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.641559 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-istiod-ca-cert\") pod \"e2e-api-b-e82419-openshift-default-786996d479-wgfzt\" (UID: \"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a\") " pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" Jul 06 20:01:23.641594 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.641587 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-workload-socket\") pod \"e2e-api-b-e82419-openshift-default-786996d479-wgfzt\" (UID: \"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a\") " pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" Jul 06 20:01:23.641809 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.641618 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-credential-socket\") pod \"e2e-api-b-e82419-openshift-default-786996d479-wgfzt\" (UID: \"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a\") " pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" Jul 06 20:01:23.641809 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.641682 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zrfk\" (UniqueName: \"kubernetes.io/projected/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-kube-api-access-4zrfk\") pod \"e2e-api-b-e82419-openshift-default-786996d479-wgfzt\" (UID: \"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a\") " pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" Jul 06 20:01:23.641809 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.641705 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-istio-podinfo\") pod \"e2e-api-b-e82419-openshift-default-786996d479-wgfzt\" (UID: \"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a\") " pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" Jul 06 20:01:23.641809 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.641743 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-istio-token\") pod \"e2e-api-b-e82419-openshift-default-786996d479-wgfzt\" (UID: \"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a\") " pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" Jul 06 20:01:23.641992 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.641827 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-istio-data\") pod \"e2e-api-b-e82419-openshift-default-786996d479-wgfzt\" (UID: \"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a\") " pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" Jul 06 20:01:23.641992 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.641856 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-workload-certs\") pod \"e2e-api-b-e82419-openshift-default-786996d479-wgfzt\" (UID: \"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a\") " pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" Jul 06 20:01:23.641992 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.641947 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-credential-socket\") pod \"e2e-api-b-e82419-openshift-default-786996d479-wgfzt\" (UID: \"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a\") " pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" Jul 06 20:01:23.642126 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.641993 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-workload-socket\") pod \"e2e-api-b-e82419-openshift-default-786996d479-wgfzt\" (UID: \"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a\") " pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" Jul 06 20:01:23.642613 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.642591 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-istiod-ca-cert\") pod \"e2e-api-b-e82419-openshift-default-786996d479-wgfzt\" (UID: \"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a\") " pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" Jul 06 20:01:23.643811 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.643786 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-istio-envoy\") pod \"e2e-api-b-e82419-openshift-default-786996d479-wgfzt\" (UID: \"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a\") " pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" Jul 06 20:01:23.644241 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.644223 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-istio-podinfo\") pod \"e2e-api-b-e82419-openshift-default-786996d479-wgfzt\" (UID: \"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a\") " pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" Jul 06 20:01:23.654170 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.654150 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zrfk\" (UniqueName: \"kubernetes.io/projected/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-kube-api-access-4zrfk\") pod \"e2e-api-b-e82419-openshift-default-786996d479-wgfzt\" (UID: \"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a\") " pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" Jul 06 20:01:23.654170 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.654156 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-istio-token\") pod \"e2e-api-b-e82419-openshift-default-786996d479-wgfzt\" (UID: \"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a\") " pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" Jul 06 20:01:23.780103 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.780049 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" Jul 06 20:01:23.934534 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.934510 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt"] Jul 06 20:01:23.937477 ip-10-0-135-23 kubenswrapper[2575]: W0706 20:01:23.937440 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8ac84d3_ed5d_4c73_9c10_0f688c5dc79a.slice/crio-3a87ef7d63559cecff14a2875356fb224309da85b2f0471d2cafea46e990427a WatchSource:0}: Error finding container 3a87ef7d63559cecff14a2875356fb224309da85b2f0471d2cafea46e990427a: Status 404 returned error can't find the container with id 3a87ef7d63559cecff14a2875356fb224309da85b2f0471d2cafea46e990427a Jul 06 20:01:23.939713 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.939679 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:01:23.939782 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.939748 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:01:23.939782 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:23.939774 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:01:24.360338 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:24.360301 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" event={"ID":"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a","Type":"ContainerStarted","Data":"0a4d6c1310c43433f625243eaa8974096285791a8a40b0a0191fe83b202475c3"} Jul 06 20:01:24.360541 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:24.360357 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" event={"ID":"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a","Type":"ContainerStarted","Data":"3a87ef7d63559cecff14a2875356fb224309da85b2f0471d2cafea46e990427a"} Jul 06 20:01:24.381868 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:24.381816 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" podStartSLOduration=1.381796979 podStartE2EDuration="1.381796979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:01:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:01:24.380445512 +0000 UTC m=+2093.200534223" watchObservedRunningTime="2026-07-06 20:01:24.381796979 +0000 UTC m=+2093.201885689" Jul 06 20:01:24.781149 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:24.781072 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" Jul 06 20:01:24.782472 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:24.782449 2575 patch_prober.go:28] interesting pod/e2e-api-b-e82419-openshift-default-786996d479-wgfzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Jul 06 20:01:24.782614 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:24.782495 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" podUID="a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Jul 06 20:01:25.783813 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:25.783785 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" Jul 06 20:01:26.375047 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:26.375009 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" Jul 06 20:01:26.375991 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:26.375965 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" Jul 06 20:01:41.376021 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:41.375983 2575 patch_prober.go:28] interesting pod/e2e-api-b-e82419-openshift-default-786996d479-wgfzt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Jul 06 20:01:41.376445 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:41.376038 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" podUID="a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Jul 06 20:01:41.470444 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:41.470419 2575 generic.go:356] "Generic (PLEG): container finished" podID="a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a" containerID="0a4d6c1310c43433f625243eaa8974096285791a8a40b0a0191fe83b202475c3" exitCode=0 Jul 06 20:01:41.470625 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:41.470501 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" event={"ID":"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a","Type":"ContainerDied","Data":"0a4d6c1310c43433f625243eaa8974096285791a8a40b0a0191fe83b202475c3"} Jul 06 20:01:41.470968 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:41.470952 2575 scope.go:117] "RemoveContainer" containerID="0a4d6c1310c43433f625243eaa8974096285791a8a40b0a0191fe83b202475c3" Jul 06 20:01:41.472146 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:41.472116 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:01:41.472245 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:41.472215 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:01:41.472414 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:41.472249 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:01:42.483979 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:42.483937 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" event={"ID":"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a","Type":"ContainerStarted","Data":"c5cbc50a373a25d2eae55a8d996b21701b74abdea9f85bfb49c5d2acbb40f550"} Jul 06 20:01:42.780866 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:42.780777 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" Jul 06 20:01:42.781267 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:42.781145 2575 patch_prober.go:28] interesting pod/e2e-api-b-e82419-openshift-default-786996d479-wgfzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Jul 06 20:01:42.781456 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:42.781429 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" podUID="a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Jul 06 20:01:43.780999 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:43.780958 2575 patch_prober.go:28] interesting pod/e2e-api-b-e82419-openshift-default-786996d479-wgfzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Jul 06 20:01:43.781482 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:43.781026 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" podUID="a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Jul 06 20:01:44.780916 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:44.780880 2575 patch_prober.go:28] interesting pod/e2e-api-b-e82419-openshift-default-786996d479-wgfzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Jul 06 20:01:44.781229 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:44.781200 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" podUID="a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Jul 06 20:01:45.781200 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:45.780745 2575 patch_prober.go:28] interesting pod/e2e-api-b-e82419-openshift-default-786996d479-wgfzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Jul 06 20:01:45.781200 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:45.780808 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" podUID="a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Jul 06 20:01:46.784911 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:46.784885 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" Jul 06 20:01:46.785319 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:46.785294 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" Jul 06 20:01:46.786040 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:46.786017 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" Jul 06 20:01:54.073200 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:54.072457 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt"] Jul 06 20:01:54.073200 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:54.072695 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" podUID="a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a" containerName="istio-proxy" containerID="cri-o://c5cbc50a373a25d2eae55a8d996b21701b74abdea9f85bfb49c5d2acbb40f550" gracePeriod=30 Jul 06 20:01:59.342760 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:59.342704 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" Jul 06 20:01:59.455291 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:59.455267 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-istio-data\") pod \"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a\" (UID: \"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a\") " Jul 06 20:01:59.455437 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:59.455296 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-istiod-ca-cert\") pod \"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a\" (UID: \"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a\") " Jul 06 20:01:59.455437 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:59.455333 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-workload-socket\") pod \"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a\" (UID: \"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a\") " Jul 06 20:01:59.455437 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:59.455353 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-istio-podinfo\") pod \"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a\" (UID: \"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a\") " Jul 06 20:01:59.455437 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:59.455377 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-credential-socket\") pod \"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a\" (UID: \"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a\") " Jul 06 20:01:59.455437 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:59.455409 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-istio-token\") pod \"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a\" (UID: \"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a\") " Jul 06 20:01:59.455699 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:59.455573 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-workload-certs\") pod \"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a\" (UID: \"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a\") " Jul 06 20:01:59.455699 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:59.455622 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a" (UID: "a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:01:59.455699 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:59.455672 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-istio-envoy\") pod \"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a\" (UID: \"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a\") " Jul 06 20:01:59.455699 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:59.455686 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a" (UID: "a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:01:59.455910 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:59.455694 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a" (UID: "a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:01:59.455910 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:59.455710 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zrfk\" (UniqueName: \"kubernetes.io/projected/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-kube-api-access-4zrfk\") pod \"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a\" (UID: \"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a\") " Jul 06 20:01:59.455910 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:59.455855 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a" (UID: "a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:01:59.456126 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:59.456099 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-istiod-ca-cert\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 20:01:59.456126 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:59.456122 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-workload-socket\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 20:01:59.456264 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:59.456135 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-credential-socket\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 20:01:59.456264 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:59.456147 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-workload-certs\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 20:01:59.456264 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:59.456232 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-istio-data" (OuterVolumeSpecName: "istio-data") pod "a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a" (UID: "a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:01:59.457601 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:59.457576 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a" (UID: "a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 20:01:59.457699 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:59.457576 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a" (UID: "a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:01:59.457699 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:59.457625 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-istio-token" (OuterVolumeSpecName: "istio-token") pod "a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a" (UID: "a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:01:59.457804 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:59.457747 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-kube-api-access-4zrfk" (OuterVolumeSpecName: "kube-api-access-4zrfk") pod "a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a" (UID: "a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a"). InnerVolumeSpecName "kube-api-access-4zrfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:01:59.557403 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:59.557384 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-istio-podinfo\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 20:01:59.557403 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:59.557403 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-istio-token\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 20:01:59.557521 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:59.557412 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-istio-envoy\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 20:01:59.557521 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:59.557420 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zrfk\" (UniqueName: \"kubernetes.io/projected/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-kube-api-access-4zrfk\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 20:01:59.557521 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:59.557427 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a-istio-data\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 20:01:59.605487 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:59.605420 2575 generic.go:356] "Generic (PLEG): container finished" podID="a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a" containerID="c5cbc50a373a25d2eae55a8d996b21701b74abdea9f85bfb49c5d2acbb40f550" exitCode=0 Jul 06 20:01:59.605487 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:59.605463 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" Jul 06 20:01:59.605487 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:59.605468 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" event={"ID":"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a","Type":"ContainerDied","Data":"c5cbc50a373a25d2eae55a8d996b21701b74abdea9f85bfb49c5d2acbb40f550"} Jul 06 20:01:59.605682 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:59.605521 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt" event={"ID":"a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a","Type":"ContainerDied","Data":"3a87ef7d63559cecff14a2875356fb224309da85b2f0471d2cafea46e990427a"} Jul 06 20:01:59.605682 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:59.605552 2575 scope.go:117] "RemoveContainer" containerID="c5cbc50a373a25d2eae55a8d996b21701b74abdea9f85bfb49c5d2acbb40f550" Jul 06 20:01:59.615444 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:59.615430 2575 scope.go:117] "RemoveContainer" containerID="0a4d6c1310c43433f625243eaa8974096285791a8a40b0a0191fe83b202475c3" Jul 06 20:01:59.623736 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:59.623718 2575 scope.go:117] "RemoveContainer" containerID="c5cbc50a373a25d2eae55a8d996b21701b74abdea9f85bfb49c5d2acbb40f550" Jul 06 20:01:59.623962 ip-10-0-135-23 kubenswrapper[2575]: E0706 20:01:59.623942 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5cbc50a373a25d2eae55a8d996b21701b74abdea9f85bfb49c5d2acbb40f550\": container with ID starting with c5cbc50a373a25d2eae55a8d996b21701b74abdea9f85bfb49c5d2acbb40f550 not found: ID does not exist" containerID="c5cbc50a373a25d2eae55a8d996b21701b74abdea9f85bfb49c5d2acbb40f550" Jul 06 20:01:59.624015 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:59.623969 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5cbc50a373a25d2eae55a8d996b21701b74abdea9f85bfb49c5d2acbb40f550"} err="failed to get container status \"c5cbc50a373a25d2eae55a8d996b21701b74abdea9f85bfb49c5d2acbb40f550\": rpc error: code = NotFound desc = could not find container \"c5cbc50a373a25d2eae55a8d996b21701b74abdea9f85bfb49c5d2acbb40f550\": container with ID starting with c5cbc50a373a25d2eae55a8d996b21701b74abdea9f85bfb49c5d2acbb40f550 not found: ID does not exist" Jul 06 20:01:59.624015 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:59.623987 2575 scope.go:117] "RemoveContainer" containerID="0a4d6c1310c43433f625243eaa8974096285791a8a40b0a0191fe83b202475c3" Jul 06 20:01:59.624231 ip-10-0-135-23 kubenswrapper[2575]: E0706 20:01:59.624212 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a4d6c1310c43433f625243eaa8974096285791a8a40b0a0191fe83b202475c3\": container with ID starting with 0a4d6c1310c43433f625243eaa8974096285791a8a40b0a0191fe83b202475c3 not found: ID does not exist" containerID="0a4d6c1310c43433f625243eaa8974096285791a8a40b0a0191fe83b202475c3" Jul 06 20:01:59.624289 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:59.624236 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a4d6c1310c43433f625243eaa8974096285791a8a40b0a0191fe83b202475c3"} err="failed to get container status \"0a4d6c1310c43433f625243eaa8974096285791a8a40b0a0191fe83b202475c3\": rpc error: code = NotFound desc = could not find container \"0a4d6c1310c43433f625243eaa8974096285791a8a40b0a0191fe83b202475c3\": container with ID starting with 0a4d6c1310c43433f625243eaa8974096285791a8a40b0a0191fe83b202475c3 not found: ID does not exist" Jul 06 20:01:59.631792 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:59.631772 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt"] Jul 06 20:01:59.634526 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:59.634509 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e82419-openshift-default-786996d479-wgfzt"] Jul 06 20:01:59.801803 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:01:59.801774 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a" path="/var/lib/kubelet/pods/a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a/volumes" Jul 06 20:02:03.059026 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.058994 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8"] Jul 06 20:02:03.059705 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.059681 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a" containerName="istio-proxy" Jul 06 20:02:03.059705 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.059703 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a" containerName="istio-proxy" Jul 06 20:02:03.059950 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.059729 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30d567b7-e41c-4528-b99f-b2f13dd07845" containerName="istio-proxy" Jul 06 20:02:03.059950 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.059738 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="30d567b7-e41c-4528-b99f-b2f13dd07845" containerName="istio-proxy" Jul 06 20:02:03.060068 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.059990 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a" containerName="istio-proxy" Jul 06 20:02:03.060068 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.060011 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a" containerName="istio-proxy" Jul 06 20:02:03.063621 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.063601 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" Jul 06 20:02:03.066496 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.066470 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-04d79c-openshift-default-dockercfg-bkpxr\"" Jul 06 20:02:03.066626 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.066531 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 06 20:02:03.076375 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.076350 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8"] Jul 06 20:02:03.188881 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.188854 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63c47bc1-d505-4229-b457-75fbc89323a1-istio-envoy\") pod \"e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8\" (UID: \"63c47bc1-d505-4229-b457-75fbc89323a1\") " pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" Jul 06 20:02:03.189006 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.188892 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63c47bc1-d505-4229-b457-75fbc89323a1-istio-data\") pod \"e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8\" (UID: \"63c47bc1-d505-4229-b457-75fbc89323a1\") " pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" Jul 06 20:02:03.189006 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.188914 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63c47bc1-d505-4229-b457-75fbc89323a1-istio-token\") pod \"e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8\" (UID: \"63c47bc1-d505-4229-b457-75fbc89323a1\") " pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" Jul 06 20:02:03.189006 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.188973 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c47bc1-d505-4229-b457-75fbc89323a1-credential-socket\") pod \"e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8\" (UID: \"63c47bc1-d505-4229-b457-75fbc89323a1\") " pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" Jul 06 20:02:03.189113 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.189044 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c47bc1-d505-4229-b457-75fbc89323a1-workload-socket\") pod \"e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8\" (UID: \"63c47bc1-d505-4229-b457-75fbc89323a1\") " pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" Jul 06 20:02:03.189113 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.189078 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63c47bc1-d505-4229-b457-75fbc89323a1-istiod-ca-cert\") pod \"e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8\" (UID: \"63c47bc1-d505-4229-b457-75fbc89323a1\") " pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" Jul 06 20:02:03.189113 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.189099 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv8jc\" (UniqueName: \"kubernetes.io/projected/63c47bc1-d505-4229-b457-75fbc89323a1-kube-api-access-wv8jc\") pod \"e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8\" (UID: \"63c47bc1-d505-4229-b457-75fbc89323a1\") " pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" Jul 06 20:02:03.189228 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.189125 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63c47bc1-d505-4229-b457-75fbc89323a1-workload-certs\") pod \"e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8\" (UID: \"63c47bc1-d505-4229-b457-75fbc89323a1\") " pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" Jul 06 20:02:03.189228 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.189207 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63c47bc1-d505-4229-b457-75fbc89323a1-istio-podinfo\") pod \"e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8\" (UID: \"63c47bc1-d505-4229-b457-75fbc89323a1\") " pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" Jul 06 20:02:03.290148 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.290123 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv8jc\" (UniqueName: \"kubernetes.io/projected/63c47bc1-d505-4229-b457-75fbc89323a1-kube-api-access-wv8jc\") pod \"e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8\" (UID: \"63c47bc1-d505-4229-b457-75fbc89323a1\") " pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" Jul 06 20:02:03.290265 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.290156 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63c47bc1-d505-4229-b457-75fbc89323a1-workload-certs\") pod \"e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8\" (UID: \"63c47bc1-d505-4229-b457-75fbc89323a1\") " pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" Jul 06 20:02:03.290265 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.290222 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63c47bc1-d505-4229-b457-75fbc89323a1-istio-podinfo\") pod \"e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8\" (UID: \"63c47bc1-d505-4229-b457-75fbc89323a1\") " pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" Jul 06 20:02:03.290265 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.290260 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63c47bc1-d505-4229-b457-75fbc89323a1-istio-envoy\") pod \"e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8\" (UID: \"63c47bc1-d505-4229-b457-75fbc89323a1\") " pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" Jul 06 20:02:03.290428 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.290401 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63c47bc1-d505-4229-b457-75fbc89323a1-istio-data\") pod \"e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8\" (UID: \"63c47bc1-d505-4229-b457-75fbc89323a1\") " pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" Jul 06 20:02:03.290486 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.290449 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63c47bc1-d505-4229-b457-75fbc89323a1-istio-token\") pod \"e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8\" (UID: \"63c47bc1-d505-4229-b457-75fbc89323a1\") " pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" Jul 06 20:02:03.290558 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.290534 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c47bc1-d505-4229-b457-75fbc89323a1-credential-socket\") pod \"e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8\" (UID: \"63c47bc1-d505-4229-b457-75fbc89323a1\") " pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" Jul 06 20:02:03.290613 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.290597 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63c47bc1-d505-4229-b457-75fbc89323a1-workload-certs\") pod \"e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8\" (UID: \"63c47bc1-d505-4229-b457-75fbc89323a1\") " pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" Jul 06 20:02:03.290668 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.290637 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c47bc1-d505-4229-b457-75fbc89323a1-workload-socket\") pod \"e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8\" (UID: \"63c47bc1-d505-4229-b457-75fbc89323a1\") " pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" Jul 06 20:02:03.290721 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.290681 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63c47bc1-d505-4229-b457-75fbc89323a1-istio-data\") pod \"e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8\" (UID: \"63c47bc1-d505-4229-b457-75fbc89323a1\") " pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" Jul 06 20:02:03.290764 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.290727 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63c47bc1-d505-4229-b457-75fbc89323a1-istiod-ca-cert\") pod \"e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8\" (UID: \"63c47bc1-d505-4229-b457-75fbc89323a1\") " pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" Jul 06 20:02:03.290876 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.290855 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c47bc1-d505-4229-b457-75fbc89323a1-credential-socket\") pod \"e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8\" (UID: \"63c47bc1-d505-4229-b457-75fbc89323a1\") " pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" Jul 06 20:02:03.290929 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.290915 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c47bc1-d505-4229-b457-75fbc89323a1-workload-socket\") pod \"e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8\" (UID: \"63c47bc1-d505-4229-b457-75fbc89323a1\") " pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" Jul 06 20:02:03.291267 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.291251 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63c47bc1-d505-4229-b457-75fbc89323a1-istiod-ca-cert\") pod \"e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8\" (UID: \"63c47bc1-d505-4229-b457-75fbc89323a1\") " pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" Jul 06 20:02:03.292530 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.292513 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63c47bc1-d505-4229-b457-75fbc89323a1-istio-envoy\") pod \"e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8\" (UID: \"63c47bc1-d505-4229-b457-75fbc89323a1\") " pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" Jul 06 20:02:03.292651 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.292636 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63c47bc1-d505-4229-b457-75fbc89323a1-istio-podinfo\") pod \"e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8\" (UID: \"63c47bc1-d505-4229-b457-75fbc89323a1\") " pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" Jul 06 20:02:03.300253 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.300236 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv8jc\" (UniqueName: \"kubernetes.io/projected/63c47bc1-d505-4229-b457-75fbc89323a1-kube-api-access-wv8jc\") pod \"e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8\" (UID: \"63c47bc1-d505-4229-b457-75fbc89323a1\") " pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" Jul 06 20:02:03.300502 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.300483 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63c47bc1-d505-4229-b457-75fbc89323a1-istio-token\") pod \"e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8\" (UID: \"63c47bc1-d505-4229-b457-75fbc89323a1\") " pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" Jul 06 20:02:03.378302 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.378282 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" Jul 06 20:02:03.525497 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.525467 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8"] Jul 06 20:02:03.526960 ip-10-0-135-23 kubenswrapper[2575]: W0706 20:02:03.526935 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63c47bc1_d505_4229_b457_75fbc89323a1.slice/crio-9e399a2fefdecc561333df2e043ae7a581cb0a26279280ba56141ae14c2988f7 WatchSource:0}: Error finding container 9e399a2fefdecc561333df2e043ae7a581cb0a26279280ba56141ae14c2988f7: Status 404 returned error can't find the container with id 9e399a2fefdecc561333df2e043ae7a581cb0a26279280ba56141ae14c2988f7 Jul 06 20:02:03.529308 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.529274 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:02:03.529418 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.529354 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:02:03.529418 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.529402 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:02:03.634834 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.634752 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" event={"ID":"63c47bc1-d505-4229-b457-75fbc89323a1","Type":"ContainerStarted","Data":"2ed2320b29ee2c519e77ad449f537f580a92a3ef8c291c35f4da2e5fe1b2d9d6"} Jul 06 20:02:03.634834 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.634825 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" event={"ID":"63c47bc1-d505-4229-b457-75fbc89323a1","Type":"ContainerStarted","Data":"9e399a2fefdecc561333df2e043ae7a581cb0a26279280ba56141ae14c2988f7"} Jul 06 20:02:03.662861 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:03.662775 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" podStartSLOduration=0.662755956 podStartE2EDuration="662.755956ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:02:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:02:03.662473859 +0000 UTC m=+2132.482562826" watchObservedRunningTime="2026-07-06 20:02:03.662755956 +0000 UTC m=+2132.482844666" Jul 06 20:02:04.378398 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:04.378368 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" Jul 06 20:02:04.383292 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:04.383271 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" Jul 06 20:02:04.641204 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:04.641123 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" Jul 06 20:02:04.641988 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:04.641973 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" Jul 06 20:02:19.642947 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:19.642913 2575 patch_prober.go:28] interesting pod/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Jul 06 20:02:19.643400 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:19.642970 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" podUID="63c47bc1-d505-4229-b457-75fbc89323a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Jul 06 20:02:19.741381 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:19.741351 2575 generic.go:356] "Generic (PLEG): container finished" podID="63c47bc1-d505-4229-b457-75fbc89323a1" containerID="2ed2320b29ee2c519e77ad449f537f580a92a3ef8c291c35f4da2e5fe1b2d9d6" exitCode=0 Jul 06 20:02:19.741533 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:19.741427 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" event={"ID":"63c47bc1-d505-4229-b457-75fbc89323a1","Type":"ContainerDied","Data":"2ed2320b29ee2c519e77ad449f537f580a92a3ef8c291c35f4da2e5fe1b2d9d6"} Jul 06 20:02:19.741810 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:19.741793 2575 scope.go:117] "RemoveContainer" containerID="2ed2320b29ee2c519e77ad449f537f580a92a3ef8c291c35f4da2e5fe1b2d9d6" Jul 06 20:02:19.742855 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:19.742823 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:02:19.742941 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:19.742909 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:02:19.742977 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:19.742956 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:02:20.750282 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:20.750246 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" event={"ID":"63c47bc1-d505-4229-b457-75fbc89323a1","Type":"ContainerStarted","Data":"5fce488ea065568d4d879b346832e8ebfe9492f103fb97de96ece843a03401b2"} Jul 06 20:02:21.379354 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:21.379321 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" Jul 06 20:02:21.379708 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:21.379664 2575 patch_prober.go:28] interesting pod/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Jul 06 20:02:21.379947 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:21.379909 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" podUID="63c47bc1-d505-4229-b457-75fbc89323a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Jul 06 20:02:22.378935 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:22.378899 2575 patch_prober.go:28] interesting pod/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Jul 06 20:02:22.379987 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:22.379956 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" podUID="63c47bc1-d505-4229-b457-75fbc89323a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Jul 06 20:02:23.378789 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:23.378706 2575 patch_prober.go:28] interesting pod/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Jul 06 20:02:23.379038 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:23.379011 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" podUID="63c47bc1-d505-4229-b457-75fbc89323a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Jul 06 20:02:24.382565 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:24.382539 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" Jul 06 20:02:24.383006 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:24.382965 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" Jul 06 20:02:24.383606 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:02:24.383582 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" Jul 06 20:06:32.052120 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:06:32.052086 2575 scope.go:117] "RemoveContainer" containerID="1cb852d768262e425edd05c2798df801d052b538093772de3310a6cef576d124" Jul 06 20:06:32.061394 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:06:32.061373 2575 scope.go:117] "RemoveContainer" containerID="c6a2e1094ec55d6c8d6e70ffcf0eba801bbee6af32dc1533f193b6b385a5b593" Jul 06 20:08:49.763148 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:49.763046 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8"] Jul 06 20:08:49.763757 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:49.763358 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" podUID="63c47bc1-d505-4229-b457-75fbc89323a1" containerName="istio-proxy" containerID="cri-o://5fce488ea065568d4d879b346832e8ebfe9492f103fb97de96ece843a03401b2" gracePeriod=30 Jul 06 20:08:54.384962 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:54.384922 2575 patch_prober.go:28] interesting pod/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 20:08:54.385393 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:54.385000 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" podUID="63c47bc1-d505-4229-b457-75fbc89323a1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:08:55.029063 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:55.028996 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" Jul 06 20:08:55.132042 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:55.132001 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c47bc1-d505-4229-b457-75fbc89323a1-credential-socket\") pod \"63c47bc1-d505-4229-b457-75fbc89323a1\" (UID: \"63c47bc1-d505-4229-b457-75fbc89323a1\") " Jul 06 20:08:55.132042 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:55.132048 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63c47bc1-d505-4229-b457-75fbc89323a1-istio-envoy\") pod \"63c47bc1-d505-4229-b457-75fbc89323a1\" (UID: \"63c47bc1-d505-4229-b457-75fbc89323a1\") " Jul 06 20:08:55.132330 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:55.132079 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c47bc1-d505-4229-b457-75fbc89323a1-workload-socket\") pod \"63c47bc1-d505-4229-b457-75fbc89323a1\" (UID: \"63c47bc1-d505-4229-b457-75fbc89323a1\") " Jul 06 20:08:55.132330 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:55.132143 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63c47bc1-d505-4229-b457-75fbc89323a1-istio-podinfo\") pod \"63c47bc1-d505-4229-b457-75fbc89323a1\" (UID: \"63c47bc1-d505-4229-b457-75fbc89323a1\") " Jul 06 20:08:55.132330 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:55.132197 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63c47bc1-d505-4229-b457-75fbc89323a1-istio-token\") pod \"63c47bc1-d505-4229-b457-75fbc89323a1\" (UID: \"63c47bc1-d505-4229-b457-75fbc89323a1\") " Jul 06 20:08:55.132330 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:55.132221 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63c47bc1-d505-4229-b457-75fbc89323a1-istiod-ca-cert\") pod \"63c47bc1-d505-4229-b457-75fbc89323a1\" (UID: \"63c47bc1-d505-4229-b457-75fbc89323a1\") " Jul 06 20:08:55.132330 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:55.132281 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63c47bc1-d505-4229-b457-75fbc89323a1-workload-certs\") pod \"63c47bc1-d505-4229-b457-75fbc89323a1\" (UID: \"63c47bc1-d505-4229-b457-75fbc89323a1\") " Jul 06 20:08:55.132609 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:55.132332 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wv8jc\" (UniqueName: \"kubernetes.io/projected/63c47bc1-d505-4229-b457-75fbc89323a1-kube-api-access-wv8jc\") pod \"63c47bc1-d505-4229-b457-75fbc89323a1\" (UID: \"63c47bc1-d505-4229-b457-75fbc89323a1\") " Jul 06 20:08:55.132609 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:55.132348 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c47bc1-d505-4229-b457-75fbc89323a1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "63c47bc1-d505-4229-b457-75fbc89323a1" (UID: "63c47bc1-d505-4229-b457-75fbc89323a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:08:55.132609 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:55.132393 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63c47bc1-d505-4229-b457-75fbc89323a1-istio-data\") pod \"63c47bc1-d505-4229-b457-75fbc89323a1\" (UID: \"63c47bc1-d505-4229-b457-75fbc89323a1\") " Jul 06 20:08:55.132609 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:55.132588 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c47bc1-d505-4229-b457-75fbc89323a1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "63c47bc1-d505-4229-b457-75fbc89323a1" (UID: "63c47bc1-d505-4229-b457-75fbc89323a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:08:55.132836 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:55.132651 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63c47bc1-d505-4229-b457-75fbc89323a1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "63c47bc1-d505-4229-b457-75fbc89323a1" (UID: "63c47bc1-d505-4229-b457-75fbc89323a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:08:55.132896 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:55.132848 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c47bc1-d505-4229-b457-75fbc89323a1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "63c47bc1-d505-4229-b457-75fbc89323a1" (UID: "63c47bc1-d505-4229-b457-75fbc89323a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:08:55.132957 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:55.132905 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c47bc1-d505-4229-b457-75fbc89323a1-credential-socket\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 20:08:55.132957 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:55.132924 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63c47bc1-d505-4229-b457-75fbc89323a1-istiod-ca-cert\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 20:08:55.132957 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:55.132938 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63c47bc1-d505-4229-b457-75fbc89323a1-workload-certs\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 20:08:55.133465 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:55.133437 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c47bc1-d505-4229-b457-75fbc89323a1-istio-data" (OuterVolumeSpecName: "istio-data") pod "63c47bc1-d505-4229-b457-75fbc89323a1" (UID: "63c47bc1-d505-4229-b457-75fbc89323a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:08:55.134593 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:55.134562 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c47bc1-d505-4229-b457-75fbc89323a1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "63c47bc1-d505-4229-b457-75fbc89323a1" (UID: "63c47bc1-d505-4229-b457-75fbc89323a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:08:55.134593 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:55.134581 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63c47bc1-d505-4229-b457-75fbc89323a1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "63c47bc1-d505-4229-b457-75fbc89323a1" (UID: "63c47bc1-d505-4229-b457-75fbc89323a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 20:08:55.134593 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:55.134567 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63c47bc1-d505-4229-b457-75fbc89323a1-istio-token" (OuterVolumeSpecName: "istio-token") pod "63c47bc1-d505-4229-b457-75fbc89323a1" (UID: "63c47bc1-d505-4229-b457-75fbc89323a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:08:55.134762 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:55.134578 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63c47bc1-d505-4229-b457-75fbc89323a1-kube-api-access-wv8jc" (OuterVolumeSpecName: "kube-api-access-wv8jc") pod "63c47bc1-d505-4229-b457-75fbc89323a1" (UID: "63c47bc1-d505-4229-b457-75fbc89323a1"). InnerVolumeSpecName "kube-api-access-wv8jc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:08:55.233839 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:55.233799 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63c47bc1-d505-4229-b457-75fbc89323a1-istio-envoy\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 20:08:55.233839 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:55.233842 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c47bc1-d505-4229-b457-75fbc89323a1-workload-socket\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 20:08:55.234060 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:55.233857 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63c47bc1-d505-4229-b457-75fbc89323a1-istio-podinfo\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 20:08:55.234060 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:55.233872 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63c47bc1-d505-4229-b457-75fbc89323a1-istio-token\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 20:08:55.234060 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:55.233885 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv8jc\" (UniqueName: \"kubernetes.io/projected/63c47bc1-d505-4229-b457-75fbc89323a1-kube-api-access-wv8jc\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 20:08:55.234060 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:55.233900 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63c47bc1-d505-4229-b457-75fbc89323a1-istio-data\") on node \"ip-10-0-135-23.ec2.internal\" DevicePath \"\"" Jul 06 20:08:55.274209 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:55.274142 2575 generic.go:356] "Generic (PLEG): container finished" podID="63c47bc1-d505-4229-b457-75fbc89323a1" containerID="5fce488ea065568d4d879b346832e8ebfe9492f103fb97de96ece843a03401b2" exitCode=0 Jul 06 20:08:55.274209 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:55.274210 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" Jul 06 20:08:55.274394 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:55.274232 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" event={"ID":"63c47bc1-d505-4229-b457-75fbc89323a1","Type":"ContainerDied","Data":"5fce488ea065568d4d879b346832e8ebfe9492f103fb97de96ece843a03401b2"} Jul 06 20:08:55.274394 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:55.274295 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8" event={"ID":"63c47bc1-d505-4229-b457-75fbc89323a1","Type":"ContainerDied","Data":"9e399a2fefdecc561333df2e043ae7a581cb0a26279280ba56141ae14c2988f7"} Jul 06 20:08:55.274394 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:55.274328 2575 scope.go:117] "RemoveContainer" containerID="5fce488ea065568d4d879b346832e8ebfe9492f103fb97de96ece843a03401b2" Jul 06 20:08:55.285524 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:55.285501 2575 scope.go:117] "RemoveContainer" containerID="2ed2320b29ee2c519e77ad449f537f580a92a3ef8c291c35f4da2e5fe1b2d9d6" Jul 06 20:08:55.295616 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:55.295596 2575 scope.go:117] "RemoveContainer" containerID="5fce488ea065568d4d879b346832e8ebfe9492f103fb97de96ece843a03401b2" Jul 06 20:08:55.295882 ip-10-0-135-23 kubenswrapper[2575]: E0706 20:08:55.295863 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fce488ea065568d4d879b346832e8ebfe9492f103fb97de96ece843a03401b2\": container with ID starting with 5fce488ea065568d4d879b346832e8ebfe9492f103fb97de96ece843a03401b2 not found: ID does not exist" containerID="5fce488ea065568d4d879b346832e8ebfe9492f103fb97de96ece843a03401b2" Jul 06 20:08:55.295940 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:55.295896 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fce488ea065568d4d879b346832e8ebfe9492f103fb97de96ece843a03401b2"} err="failed to get container status \"5fce488ea065568d4d879b346832e8ebfe9492f103fb97de96ece843a03401b2\": rpc error: code = NotFound desc = could not find container \"5fce488ea065568d4d879b346832e8ebfe9492f103fb97de96ece843a03401b2\": container with ID starting with 5fce488ea065568d4d879b346832e8ebfe9492f103fb97de96ece843a03401b2 not found: ID does not exist" Jul 06 20:08:55.295940 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:55.295920 2575 scope.go:117] "RemoveContainer" containerID="2ed2320b29ee2c519e77ad449f537f580a92a3ef8c291c35f4da2e5fe1b2d9d6" Jul 06 20:08:55.296169 ip-10-0-135-23 kubenswrapper[2575]: E0706 20:08:55.296146 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ed2320b29ee2c519e77ad449f537f580a92a3ef8c291c35f4da2e5fe1b2d9d6\": container with ID starting with 2ed2320b29ee2c519e77ad449f537f580a92a3ef8c291c35f4da2e5fe1b2d9d6 not found: ID does not exist" containerID="2ed2320b29ee2c519e77ad449f537f580a92a3ef8c291c35f4da2e5fe1b2d9d6" Jul 06 20:08:55.296252 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:55.296193 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ed2320b29ee2c519e77ad449f537f580a92a3ef8c291c35f4da2e5fe1b2d9d6"} err="failed to get container status \"2ed2320b29ee2c519e77ad449f537f580a92a3ef8c291c35f4da2e5fe1b2d9d6\": rpc error: code = NotFound desc = could not find container \"2ed2320b29ee2c519e77ad449f537f580a92a3ef8c291c35f4da2e5fe1b2d9d6\": container with ID starting with 2ed2320b29ee2c519e77ad449f537f580a92a3ef8c291c35f4da2e5fe1b2d9d6 not found: ID does not exist" Jul 06 20:08:55.302975 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:55.302948 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8"] Jul 06 20:08:55.307095 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:55.307069 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-04d79c-openshift-default-746c98b975-lczt8"] Jul 06 20:08:55.651331 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:55.651302 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-l8hsx_67cd03db-d916-4703-9f42-b5e0df241829/manager/1.log" Jul 06 20:08:55.790713 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:55.790680 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-8vh5h_1c159251-98a2-4993-af7d-aecf281fbf67/manager/1.log" Jul 06 20:08:55.796621 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:55.796592 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63c47bc1-d505-4229-b457-75fbc89323a1" path="/var/lib/kubelet/pods/63c47bc1-d505-4229-b457-75fbc89323a1/volumes" Jul 06 20:08:56.068596 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:56.068523 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722785-t92np_bc158d81-1ccd-4e69-b620-0cb53d5c98ec/cleanup/0.log" Jul 06 20:08:56.202999 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:56.202972 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722800-xdn8c_52006610-e3c3-40bb-993a-5c58e1283f26/cleanup/0.log" Jul 06 20:08:56.334818 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:56.334785 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-89b4bf89b-sxltm_5b2875d4-ab7f-4bf1-851d-a4006df17049/manager/1.log" Jul 06 20:08:56.463418 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:56.463394 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-x9n6r_d3e123fa-452d-4cb3-b12e-ff02d37e6ac9/server/0.log" Jul 06 20:08:56.594037 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:56.593950 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-qc9r4_bba14ce7-8926-47cd-bf1c-474c7d179cdf/manager/1.log" Jul 06 20:08:57.000622 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:57.000541 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5586496c87-l267l_de811e8a-5b85-4707-abe7-79107ab0a37a/copy-manifests/0.log" Jul 06 20:08:57.015615 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:57.015574 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5586496c87-l267l_de811e8a-5b85-4707-abe7-79107ab0a37a/manager/1.log" Jul 06 20:08:57.143689 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:57.143660 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-jk6zp_05eddb0a-97b3-4aca-8d15-a1f739f6d45d/postgres/0.log" Jul 06 20:08:58.036148 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:58.036116 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dbs8c_cb57645a-f2c3-4241-8929-2e058bfbffbe/util/0.log" Jul 06 20:08:58.044744 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:58.044720 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dbs8c_cb57645a-f2c3-4241-8929-2e058bfbffbe/pull/0.log" Jul 06 20:08:58.052724 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:58.052702 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dbs8c_cb57645a-f2c3-4241-8929-2e058bfbffbe/extract/0.log" Jul 06 20:08:58.178565 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:58.178539 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08d46k_f77e4ac6-70cf-4dc0-8829-740324b3f385/extract/0.log" Jul 06 20:08:58.186031 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:58.186005 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08d46k_f77e4ac6-70cf-4dc0-8829-740324b3f385/util/0.log" Jul 06 20:08:58.194665 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:58.194638 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08d46k_f77e4ac6-70cf-4dc0-8829-740324b3f385/pull/0.log" Jul 06 20:08:58.322957 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:58.322930 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gq9fm_6a90a1c3-1bff-4393-a787-f78b5e0e367f/extract/0.log" Jul 06 20:08:58.331447 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:58.331424 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gq9fm_6a90a1c3-1bff-4393-a787-f78b5e0e367f/util/0.log" Jul 06 20:08:58.338921 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:58.338896 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gq9fm_6a90a1c3-1bff-4393-a787-f78b5e0e367f/pull/0.log" Jul 06 20:08:58.461566 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:58.461534 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kjf75_3b2b2fce-db68-4292-bc62-08e5d48d2c3f/util/0.log" Jul 06 20:08:58.470026 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:58.470000 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kjf75_3b2b2fce-db68-4292-bc62-08e5d48d2c3f/pull/0.log" Jul 06 20:08:58.478040 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:58.478017 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kjf75_3b2b2fce-db68-4292-bc62-08e5d48d2c3f/extract/0.log" Jul 06 20:08:58.737273 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:58.737169 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-5cc9j_ebc5be6f-3e75-4208-b275-a4752dbf5b06/manager/1.log" Jul 06 20:08:58.862386 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:58.862361 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-8xjts_e692ea80-46aa-4916-a536-ce492c375f61/manager/2.log" Jul 06 20:08:59.140404 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:59.140372 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-f2kc9_7cafd8f9-3644-4b3a-a810-de3ebf82d7fd/registry-server/0.log" Jul 06 20:08:59.271379 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:59.271352 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-6qjzf_2f406c37-7a7b-4662-9c96-f1b16ba3b0fa/manager/0.log" Jul 06 20:08:59.519952 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:08:59.519869 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-x88hr_3a81b584-f08e-4e8d-922c-5737522ef685/manager/1.log" Jul 06 20:09:00.166002 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:00.165973 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c47f5985b-pczqp_b77b8d39-d01b-409d-bbbc-1e30d9200e1c/kube-auth-proxy/0.log" Jul 06 20:09:00.541824 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:00.541735 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-6wbdc_dd268b94-55ca-4307-beb5-29fdc2d18c78/payload-pre-processing/0.log" Jul 06 20:09:00.783057 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:00.783029 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6dcdb5648f-whl8g_677b6ba1-3252-4777-88fb-b4238918b699/router/0.log" Jul 06 20:09:01.170279 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:01.170252 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-csp7c_4355afe1-da7d-4bee-9033-1730e8b75251/main/0.log" Jul 06 20:09:01.297920 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:01.297890 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-zjfn7_4ca952cf-2db3-4a5a-9b7f-b1308373192e/main/0.log" Jul 06 20:09:06.364729 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:06.364692 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-szkqh/must-gather-x5m2r"] Jul 06 20:09:06.365434 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:06.365406 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63c47bc1-d505-4229-b457-75fbc89323a1" containerName="istio-proxy" Jul 06 20:09:06.365434 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:06.365430 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="63c47bc1-d505-4229-b457-75fbc89323a1" containerName="istio-proxy" Jul 06 20:09:06.365632 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:06.365443 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a" containerName="istio-proxy" Jul 06 20:09:06.365632 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:06.365451 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8ac84d3-ed5d-4c73-9c10-0f688c5dc79a" containerName="istio-proxy" Jul 06 20:09:06.365632 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:06.365587 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="63c47bc1-d505-4229-b457-75fbc89323a1" containerName="istio-proxy" Jul 06 20:09:06.365790 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:06.365735 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63c47bc1-d505-4229-b457-75fbc89323a1" containerName="istio-proxy" Jul 06 20:09:06.365790 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:06.365746 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="63c47bc1-d505-4229-b457-75fbc89323a1" containerName="istio-proxy" Jul 06 20:09:06.365913 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:06.365841 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="63c47bc1-d505-4229-b457-75fbc89323a1" containerName="istio-proxy" Jul 06 20:09:06.369706 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:06.369687 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-szkqh/must-gather-x5m2r" Jul 06 20:09:06.375860 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:06.375837 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-szkqh\"/\"openshift-service-ca.crt\"" Jul 06 20:09:06.376861 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:06.376685 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-szkqh\"/\"default-dockercfg-bgf4z\"" Jul 06 20:09:06.376861 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:06.376706 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-szkqh\"/\"kube-root-ca.crt\"" Jul 06 20:09:06.389702 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:06.389681 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-szkqh/must-gather-x5m2r"] Jul 06 20:09:06.440782 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:06.440759 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b1d3b8d2-a5ce-4236-9223-272b8e52bc51-must-gather-output\") pod \"must-gather-x5m2r\" (UID: \"b1d3b8d2-a5ce-4236-9223-272b8e52bc51\") " pod="openshift-must-gather-szkqh/must-gather-x5m2r" Jul 06 20:09:06.440882 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:06.440808 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nptcg\" (UniqueName: \"kubernetes.io/projected/b1d3b8d2-a5ce-4236-9223-272b8e52bc51-kube-api-access-nptcg\") pod \"must-gather-x5m2r\" (UID: \"b1d3b8d2-a5ce-4236-9223-272b8e52bc51\") " pod="openshift-must-gather-szkqh/must-gather-x5m2r" Jul 06 20:09:06.542242 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:06.542206 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b1d3b8d2-a5ce-4236-9223-272b8e52bc51-must-gather-output\") pod \"must-gather-x5m2r\" (UID: \"b1d3b8d2-a5ce-4236-9223-272b8e52bc51\") " pod="openshift-must-gather-szkqh/must-gather-x5m2r" Jul 06 20:09:06.542442 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:06.542302 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nptcg\" (UniqueName: \"kubernetes.io/projected/b1d3b8d2-a5ce-4236-9223-272b8e52bc51-kube-api-access-nptcg\") pod \"must-gather-x5m2r\" (UID: \"b1d3b8d2-a5ce-4236-9223-272b8e52bc51\") " pod="openshift-must-gather-szkqh/must-gather-x5m2r" Jul 06 20:09:06.542586 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:06.542566 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b1d3b8d2-a5ce-4236-9223-272b8e52bc51-must-gather-output\") pod \"must-gather-x5m2r\" (UID: \"b1d3b8d2-a5ce-4236-9223-272b8e52bc51\") " pod="openshift-must-gather-szkqh/must-gather-x5m2r" Jul 06 20:09:06.553269 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:06.553241 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nptcg\" (UniqueName: \"kubernetes.io/projected/b1d3b8d2-a5ce-4236-9223-272b8e52bc51-kube-api-access-nptcg\") pod \"must-gather-x5m2r\" (UID: \"b1d3b8d2-a5ce-4236-9223-272b8e52bc51\") " pod="openshift-must-gather-szkqh/must-gather-x5m2r" Jul 06 20:09:06.679416 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:06.679326 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-szkqh/must-gather-x5m2r" Jul 06 20:09:06.836172 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:06.836147 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-szkqh/must-gather-x5m2r"] Jul 06 20:09:06.838237 ip-10-0-135-23 kubenswrapper[2575]: W0706 20:09:06.838211 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1d3b8d2_a5ce_4236_9223_272b8e52bc51.slice/crio-30a6c8b9fd05d08f35e9a380cb65f8b7fcb4742c40a3bd62fe017a4b5b94068a WatchSource:0}: Error finding container 30a6c8b9fd05d08f35e9a380cb65f8b7fcb4742c40a3bd62fe017a4b5b94068a: Status 404 returned error can't find the container with id 30a6c8b9fd05d08f35e9a380cb65f8b7fcb4742c40a3bd62fe017a4b5b94068a Jul 06 20:09:06.840156 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:06.840139 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 20:09:07.355879 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:07.355844 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-szkqh/must-gather-x5m2r" event={"ID":"b1d3b8d2-a5ce-4236-9223-272b8e52bc51","Type":"ContainerStarted","Data":"30a6c8b9fd05d08f35e9a380cb65f8b7fcb4742c40a3bd62fe017a4b5b94068a"} Jul 06 20:09:08.368356 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:08.368315 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-szkqh/must-gather-x5m2r" event={"ID":"b1d3b8d2-a5ce-4236-9223-272b8e52bc51","Type":"ContainerStarted","Data":"57b44e3c266fa1e475bae819c595ae733d31d88cba95d45c92e0ec192db59e0c"} Jul 06 20:09:08.368766 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:08.368369 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-szkqh/must-gather-x5m2r" event={"ID":"b1d3b8d2-a5ce-4236-9223-272b8e52bc51","Type":"ContainerStarted","Data":"4ff15a1e66998d7c9dbe85f3be534ca7fb6b5b059288c7432c708e508e6c9158"} Jul 06 20:09:08.386541 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:08.386488 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-szkqh/must-gather-x5m2r" podStartSLOduration=1.508822333 podStartE2EDuration="2.386470029s" totalImagesPullingTime="877.647696ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:09:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:09:08.384995489 +0000 UTC m=+2557.205084202" watchObservedRunningTime="2026-07-06 20:09:08.386470029 +0000 UTC m=+2557.206558741" Jul 06 20:09:09.619442 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:09.619390 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-h7z7k_696eee70-dc27-4b63-b656-ab71bd2ebec5/global-pull-secret-syncer/0.log" Jul 06 20:09:09.711873 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:09.711840 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pgh76_556124af-4be3-4157-8b4b-70c1c1dfb6fb/konnectivity-agent/0.log" Jul 06 20:09:09.776121 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:09.776094 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-23.ec2.internal_42d465f5997f682b6201edde58cb1d3a/haproxy/0.log" Jul 06 20:09:14.426321 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:14.426290 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dbs8c_cb57645a-f2c3-4241-8929-2e058bfbffbe/extract/0.log" Jul 06 20:09:14.452621 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:14.452594 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dbs8c_cb57645a-f2c3-4241-8929-2e058bfbffbe/util/0.log" Jul 06 20:09:14.481143 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:14.481093 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dbs8c_cb57645a-f2c3-4241-8929-2e058bfbffbe/pull/0.log" Jul 06 20:09:14.512486 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:14.512461 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08d46k_f77e4ac6-70cf-4dc0-8829-740324b3f385/extract/0.log" Jul 06 20:09:14.542372 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:14.542301 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08d46k_f77e4ac6-70cf-4dc0-8829-740324b3f385/util/0.log" Jul 06 20:09:14.571593 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:14.571567 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08d46k_f77e4ac6-70cf-4dc0-8829-740324b3f385/pull/0.log" Jul 06 20:09:14.610420 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:14.610393 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gq9fm_6a90a1c3-1bff-4393-a787-f78b5e0e367f/extract/0.log" Jul 06 20:09:14.634873 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:14.634836 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gq9fm_6a90a1c3-1bff-4393-a787-f78b5e0e367f/util/0.log" Jul 06 20:09:14.668670 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:14.668624 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gq9fm_6a90a1c3-1bff-4393-a787-f78b5e0e367f/pull/0.log" Jul 06 20:09:14.704958 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:14.704879 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kjf75_3b2b2fce-db68-4292-bc62-08e5d48d2c3f/extract/0.log" Jul 06 20:09:14.732491 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:14.732406 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kjf75_3b2b2fce-db68-4292-bc62-08e5d48d2c3f/util/0.log" Jul 06 20:09:14.766645 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:14.766620 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kjf75_3b2b2fce-db68-4292-bc62-08e5d48d2c3f/pull/0.log" Jul 06 20:09:15.067770 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:15.067681 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-5cc9j_ebc5be6f-3e75-4208-b275-a4752dbf5b06/manager/1.log" Jul 06 20:09:15.071324 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:15.071296 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-5cc9j_ebc5be6f-3e75-4208-b275-a4752dbf5b06/manager/0.log" Jul 06 20:09:15.099922 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:15.099708 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-8xjts_e692ea80-46aa-4916-a536-ce492c375f61/manager/2.log" Jul 06 20:09:15.099922 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:15.099847 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-8xjts_e692ea80-46aa-4916-a536-ce492c375f61/manager/1.log" Jul 06 20:09:15.178089 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:15.178050 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-f2kc9_7cafd8f9-3644-4b3a-a810-de3ebf82d7fd/registry-server/0.log" Jul 06 20:09:15.348192 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:15.348147 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-6qjzf_2f406c37-7a7b-4662-9c96-f1b16ba3b0fa/manager/0.log" Jul 06 20:09:15.436614 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:15.436588 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-x88hr_3a81b584-f08e-4e8d-922c-5737522ef685/manager/0.log" Jul 06 20:09:15.477287 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:15.477251 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-x88hr_3a81b584-f08e-4e8d-922c-5737522ef685/manager/1.log" Jul 06 20:09:17.282998 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:17.282965 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_24edef41-b63d-4fe8-9bd9-c3f28160cde8/alertmanager/0.log" Jul 06 20:09:17.323819 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:17.323787 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_24edef41-b63d-4fe8-9bd9-c3f28160cde8/config-reloader/0.log" Jul 06 20:09:17.354445 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:17.354413 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_24edef41-b63d-4fe8-9bd9-c3f28160cde8/kube-rbac-proxy-web/0.log" Jul 06 20:09:17.383026 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:17.382996 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_24edef41-b63d-4fe8-9bd9-c3f28160cde8/kube-rbac-proxy/0.log" Jul 06 20:09:17.416244 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:17.416210 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_24edef41-b63d-4fe8-9bd9-c3f28160cde8/kube-rbac-proxy-metric/0.log" Jul 06 20:09:17.448313 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:17.448281 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_24edef41-b63d-4fe8-9bd9-c3f28160cde8/prom-label-proxy/0.log" Jul 06 20:09:17.484127 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:17.484091 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_24edef41-b63d-4fe8-9bd9-c3f28160cde8/init-config-reloader/0.log" Jul 06 20:09:17.579451 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:17.579426 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-wm4jf_12d78a18-d2c1-429d-81bb-14f3b8e4c21e/kube-state-metrics/0.log" Jul 06 20:09:17.607832 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:17.607769 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-wm4jf_12d78a18-d2c1-429d-81bb-14f3b8e4c21e/kube-rbac-proxy-main/0.log" Jul 06 20:09:17.635315 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:17.635284 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-wm4jf_12d78a18-d2c1-429d-81bb-14f3b8e4c21e/kube-rbac-proxy-self/0.log" Jul 06 20:09:17.667263 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:17.667218 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5f5769b66b-86k92_9c0394d2-076a-4bef-9198-be9301946e20/metrics-server/0.log" Jul 06 20:09:17.729791 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:17.729758 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4s6hk_f2ebcfe7-cf1d-46b8-b531-92f1502af453/node-exporter/0.log" Jul 06 20:09:17.755966 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:17.755940 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4s6hk_f2ebcfe7-cf1d-46b8-b531-92f1502af453/kube-rbac-proxy/0.log" Jul 06 20:09:17.791201 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:17.790750 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4s6hk_f2ebcfe7-cf1d-46b8-b531-92f1502af453/init-textfile/0.log" Jul 06 20:09:18.031683 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:18.031578 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-dxk9h_ee3ac022-0831-45e4-95b0-a9e1c2a1b919/kube-rbac-proxy-main/0.log" Jul 06 20:09:18.065839 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:18.065808 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-dxk9h_ee3ac022-0831-45e4-95b0-a9e1c2a1b919/kube-rbac-proxy-self/0.log" Jul 06 20:09:18.096818 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:18.096784 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-dxk9h_ee3ac022-0831-45e4-95b0-a9e1c2a1b919/openshift-state-metrics/0.log" Jul 06 20:09:18.353578 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:18.353542 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-tjmst_e6a2716f-443f-4b0a-98f4-c5a88c2785c7/prometheus-operator/0.log" Jul 06 20:09:18.377099 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:18.377071 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-tjmst_e6a2716f-443f-4b0a-98f4-c5a88c2785c7/kube-rbac-proxy/0.log" Jul 06 20:09:18.441944 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:18.441913 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6749c8c9bc-x59nv_15a01f49-5b2a-4c49-8f3b-4f23a6d48b45/telemeter-client/0.log" Jul 06 20:09:18.472721 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:18.472696 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6749c8c9bc-x59nv_15a01f49-5b2a-4c49-8f3b-4f23a6d48b45/reload/0.log" Jul 06 20:09:18.502828 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:18.502791 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6749c8c9bc-x59nv_15a01f49-5b2a-4c49-8f3b-4f23a6d48b45/kube-rbac-proxy/0.log" Jul 06 20:09:18.539015 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:18.538987 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c7cf86bbd-ck249_cf8e44b4-157b-43f6-b854-5b39e1e718c9/thanos-query/0.log" Jul 06 20:09:18.573837 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:18.573801 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c7cf86bbd-ck249_cf8e44b4-157b-43f6-b854-5b39e1e718c9/kube-rbac-proxy-web/0.log" Jul 06 20:09:18.601566 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:18.601537 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c7cf86bbd-ck249_cf8e44b4-157b-43f6-b854-5b39e1e718c9/kube-rbac-proxy/0.log" Jul 06 20:09:18.627323 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:18.627236 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c7cf86bbd-ck249_cf8e44b4-157b-43f6-b854-5b39e1e718c9/prom-label-proxy/0.log" Jul 06 20:09:18.655852 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:18.655774 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c7cf86bbd-ck249_cf8e44b4-157b-43f6-b854-5b39e1e718c9/kube-rbac-proxy-rules/0.log" Jul 06 20:09:18.681107 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:18.681076 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c7cf86bbd-ck249_cf8e44b4-157b-43f6-b854-5b39e1e718c9/kube-rbac-proxy-metrics/0.log" Jul 06 20:09:19.351630 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:19.351589 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-szkqh/perf-node-gather-daemonset-fwks9"] Jul 06 20:09:19.363028 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:19.362992 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-szkqh/perf-node-gather-daemonset-fwks9" Jul 06 20:09:19.367394 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:19.367362 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-szkqh/perf-node-gather-daemonset-fwks9"] Jul 06 20:09:19.406021 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:19.403832 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44c6d8bb-4375-487f-8560-76c3eccdd789-sys\") pod \"perf-node-gather-daemonset-fwks9\" (UID: \"44c6d8bb-4375-487f-8560-76c3eccdd789\") " pod="openshift-must-gather-szkqh/perf-node-gather-daemonset-fwks9" Jul 06 20:09:19.406021 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:19.403895 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dmqm\" (UniqueName: \"kubernetes.io/projected/44c6d8bb-4375-487f-8560-76c3eccdd789-kube-api-access-5dmqm\") pod \"perf-node-gather-daemonset-fwks9\" (UID: \"44c6d8bb-4375-487f-8560-76c3eccdd789\") " pod="openshift-must-gather-szkqh/perf-node-gather-daemonset-fwks9" Jul 06 20:09:19.406021 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:19.403941 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/44c6d8bb-4375-487f-8560-76c3eccdd789-lib-modules\") pod \"perf-node-gather-daemonset-fwks9\" (UID: \"44c6d8bb-4375-487f-8560-76c3eccdd789\") " pod="openshift-must-gather-szkqh/perf-node-gather-daemonset-fwks9" Jul 06 20:09:19.406021 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:19.404034 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/44c6d8bb-4375-487f-8560-76c3eccdd789-podres\") pod \"perf-node-gather-daemonset-fwks9\" (UID: \"44c6d8bb-4375-487f-8560-76c3eccdd789\") " pod="openshift-must-gather-szkqh/perf-node-gather-daemonset-fwks9" Jul 06 20:09:19.406021 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:19.404060 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/44c6d8bb-4375-487f-8560-76c3eccdd789-proc\") pod \"perf-node-gather-daemonset-fwks9\" (UID: \"44c6d8bb-4375-487f-8560-76c3eccdd789\") " pod="openshift-must-gather-szkqh/perf-node-gather-daemonset-fwks9" Jul 06 20:09:19.505338 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:19.505283 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44c6d8bb-4375-487f-8560-76c3eccdd789-sys\") pod \"perf-node-gather-daemonset-fwks9\" (UID: \"44c6d8bb-4375-487f-8560-76c3eccdd789\") " pod="openshift-must-gather-szkqh/perf-node-gather-daemonset-fwks9" Jul 06 20:09:19.505533 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:19.505364 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dmqm\" (UniqueName: \"kubernetes.io/projected/44c6d8bb-4375-487f-8560-76c3eccdd789-kube-api-access-5dmqm\") pod \"perf-node-gather-daemonset-fwks9\" (UID: \"44c6d8bb-4375-487f-8560-76c3eccdd789\") " pod="openshift-must-gather-szkqh/perf-node-gather-daemonset-fwks9" Jul 06 20:09:19.505533 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:19.505422 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/44c6d8bb-4375-487f-8560-76c3eccdd789-lib-modules\") pod \"perf-node-gather-daemonset-fwks9\" (UID: \"44c6d8bb-4375-487f-8560-76c3eccdd789\") " pod="openshift-must-gather-szkqh/perf-node-gather-daemonset-fwks9" Jul 06 20:09:19.505660 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:19.505535 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/44c6d8bb-4375-487f-8560-76c3eccdd789-podres\") pod \"perf-node-gather-daemonset-fwks9\" (UID: \"44c6d8bb-4375-487f-8560-76c3eccdd789\") " pod="openshift-must-gather-szkqh/perf-node-gather-daemonset-fwks9" Jul 06 20:09:19.505660 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:19.505566 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/44c6d8bb-4375-487f-8560-76c3eccdd789-proc\") pod \"perf-node-gather-daemonset-fwks9\" (UID: \"44c6d8bb-4375-487f-8560-76c3eccdd789\") " pod="openshift-must-gather-szkqh/perf-node-gather-daemonset-fwks9" Jul 06 20:09:19.505770 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:19.505703 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/44c6d8bb-4375-487f-8560-76c3eccdd789-proc\") pod \"perf-node-gather-daemonset-fwks9\" (UID: \"44c6d8bb-4375-487f-8560-76c3eccdd789\") " pod="openshift-must-gather-szkqh/perf-node-gather-daemonset-fwks9" Jul 06 20:09:19.505828 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:19.505780 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44c6d8bb-4375-487f-8560-76c3eccdd789-sys\") pod \"perf-node-gather-daemonset-fwks9\" (UID: \"44c6d8bb-4375-487f-8560-76c3eccdd789\") " pod="openshift-must-gather-szkqh/perf-node-gather-daemonset-fwks9" Jul 06 20:09:19.506060 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:19.506038 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/44c6d8bb-4375-487f-8560-76c3eccdd789-lib-modules\") pod \"perf-node-gather-daemonset-fwks9\" (UID: \"44c6d8bb-4375-487f-8560-76c3eccdd789\") " pod="openshift-must-gather-szkqh/perf-node-gather-daemonset-fwks9" Jul 06 20:09:19.506221 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:19.506169 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/44c6d8bb-4375-487f-8560-76c3eccdd789-podres\") pod \"perf-node-gather-daemonset-fwks9\" (UID: \"44c6d8bb-4375-487f-8560-76c3eccdd789\") " pod="openshift-must-gather-szkqh/perf-node-gather-daemonset-fwks9" Jul 06 20:09:19.515665 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:19.515629 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dmqm\" (UniqueName: \"kubernetes.io/projected/44c6d8bb-4375-487f-8560-76c3eccdd789-kube-api-access-5dmqm\") pod \"perf-node-gather-daemonset-fwks9\" (UID: \"44c6d8bb-4375-487f-8560-76c3eccdd789\") " pod="openshift-must-gather-szkqh/perf-node-gather-daemonset-fwks9" Jul 06 20:09:19.680586 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:19.680492 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-szkqh/perf-node-gather-daemonset-fwks9" Jul 06 20:09:19.882436 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:19.882313 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-szkqh/perf-node-gather-daemonset-fwks9"] Jul 06 20:09:20.490574 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:20.490533 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-szkqh/perf-node-gather-daemonset-fwks9" event={"ID":"44c6d8bb-4375-487f-8560-76c3eccdd789","Type":"ContainerStarted","Data":"55e4919136195d37d7322c8581421b7cceb748a1be006a2f0ec3fe1a86892360"} Jul 06 20:09:20.491069 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:20.490637 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-szkqh/perf-node-gather-daemonset-fwks9" event={"ID":"44c6d8bb-4375-487f-8560-76c3eccdd789","Type":"ContainerStarted","Data":"8a8dae6caeb0ee5b06278808204ae0d5e833f16cacf3ae0b1df6e246bea0bdf4"} Jul 06 20:09:20.491069 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:20.490830 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-szkqh/perf-node-gather-daemonset-fwks9" Jul 06 20:09:20.522672 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:20.522604 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-szkqh/perf-node-gather-daemonset-fwks9" podStartSLOduration=1.5225842699999999 podStartE2EDuration="1.52258427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:09:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:09:20.519934172 +0000 UTC m=+2569.340022907" watchObservedRunningTime="2026-07-06 20:09:20.52258427 +0000 UTC m=+2569.342672982" Jul 06 20:09:21.132825 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:21.132794 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7c9cb4cdbf-zcnzl_6f298902-626e-4071-ab20-1629b3b97359/console/0.log" Jul 06 20:09:21.169221 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:21.169172 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-rsg5c_d3f27b98-edb6-4774-a5ea-330b55f2e0ec/download-server/0.log" Jul 06 20:09:21.865756 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:21.865724 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-b8955_70caece2-9e4c-4518-9a0d-b36a8cc17d50/volume-data-source-validator/0.log" Jul 06 20:09:23.047798 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:23.047773 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-272x4_1adc5a22-7a57-4da4-9a66-0c451f2603be/dns/0.log" Jul 06 20:09:23.074045 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:23.074022 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-272x4_1adc5a22-7a57-4da4-9a66-0c451f2603be/kube-rbac-proxy/0.log" Jul 06 20:09:23.214468 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:23.214433 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2xwfj_9bd815c1-1206-431e-9d76-ba36f79833ac/dns-node-resolver/0.log" Jul 06 20:09:23.980504 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:23.980471 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q2tnp_863c8526-b822-48d4-8881-819331146578/node-ca/0.log" Jul 06 20:09:25.236264 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:25.236232 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c47f5985b-pczqp_b77b8d39-d01b-409d-bbbc-1e30d9200e1c/kube-auth-proxy/0.log" Jul 06 20:09:25.375848 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:25.375798 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-6wbdc_dd268b94-55ca-4307-beb5-29fdc2d18c78/payload-pre-processing/0.log" Jul 06 20:09:25.440314 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:25.440284 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6dcdb5648f-whl8g_677b6ba1-3252-4777-88fb-b4238918b699/router/0.log" Jul 06 20:09:26.244402 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:26.244362 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-96bb8_d0f9bf38-0857-458d-9962-51a2502b2a20/serve-healthcheck-canary/0.log" Jul 06 20:09:26.510485 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:26.510414 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-szkqh/perf-node-gather-daemonset-fwks9" Jul 06 20:09:26.899608 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:26.899570 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-5wmbj_e3efc9cc-6d76-4cde-a12a-c0f5022e1abf/insights-operator/1.log" Jul 06 20:09:26.901218 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:26.901144 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-5wmbj_e3efc9cc-6d76-4cde-a12a-c0f5022e1abf/insights-operator/0.log" Jul 06 20:09:26.932151 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:26.932118 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gnxgr_660f827d-c694-49d5-9af0-1458f46f2ffb/kube-rbac-proxy/0.log" Jul 06 20:09:26.969499 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:26.969454 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gnxgr_660f827d-c694-49d5-9af0-1458f46f2ffb/exporter/0.log" Jul 06 20:09:27.002864 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:27.002834 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gnxgr_660f827d-c694-49d5-9af0-1458f46f2ffb/extractor/0.log" Jul 06 20:09:29.732934 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:29.732900 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c47f5985b-pczqp_b77b8d39-d01b-409d-bbbc-1e30d9200e1c/kube-auth-proxy/0.log" Jul 06 20:09:29.757296 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:29.757265 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-l8hsx_67cd03db-d916-4703-9f42-b5e0df241829/manager/0.log" Jul 06 20:09:29.757838 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:29.757819 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-l8hsx_67cd03db-d916-4703-9f42-b5e0df241829/manager/1.log" Jul 06 20:09:29.825238 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:29.825200 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-8vh5h_1c159251-98a2-4993-af7d-aecf281fbf67/manager/0.log" Jul 06 20:09:29.839689 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:29.839655 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-6wbdc_dd268b94-55ca-4307-beb5-29fdc2d18c78/payload-pre-processing/0.log" Jul 06 20:09:29.890775 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:29.890738 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6dcdb5648f-whl8g_677b6ba1-3252-4777-88fb-b4238918b699/router/0.log" Jul 06 20:09:29.946173 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:29.946123 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-8vh5h_1c159251-98a2-4993-af7d-aecf281fbf67/manager/1.log" Jul 06 20:09:30.011328 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:30.011250 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722785-t92np_bc158d81-1ccd-4e69-b620-0cb53d5c98ec/cleanup/0.log" Jul 06 20:09:30.038590 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:30.038560 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722800-xdn8c_52006610-e3c3-40bb-993a-5c58e1283f26/cleanup/0.log" Jul 06 20:09:30.082595 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:30.082562 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-89b4bf89b-sxltm_5b2875d4-ab7f-4bf1-851d-a4006df17049/manager/0.log" Jul 06 20:09:30.167814 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:30.167765 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-89b4bf89b-sxltm_5b2875d4-ab7f-4bf1-851d-a4006df17049/manager/1.log" Jul 06 20:09:30.193540 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:30.193510 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-x9n6r_d3e123fa-452d-4cb3-b12e-ff02d37e6ac9/server/0.log" Jul 06 20:09:30.273676 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:30.273582 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-qc9r4_bba14ce7-8926-47cd-bf1c-474c7d179cdf/manager/0.log" Jul 06 20:09:30.274044 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:30.274025 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-qc9r4_bba14ce7-8926-47cd-bf1c-474c7d179cdf/manager/1.log" Jul 06 20:09:30.556161 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:30.556075 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5586496c87-l267l_de811e8a-5b85-4707-abe7-79107ab0a37a/manager/0.log" Jul 06 20:09:30.572441 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:30.572405 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5586496c87-l267l_de811e8a-5b85-4707-abe7-79107ab0a37a/manager/1.log" Jul 06 20:09:30.596305 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:30.596278 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5586496c87-l267l_de811e8a-5b85-4707-abe7-79107ab0a37a/copy-manifests/0.log" Jul 06 20:09:30.626560 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:30.626531 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-jk6zp_05eddb0a-97b3-4aca-8d15-a1f739f6d45d/postgres/0.log" Jul 06 20:09:31.251283 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:31.251258 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c47f5985b-pczqp_b77b8d39-d01b-409d-bbbc-1e30d9200e1c/kube-auth-proxy/0.log" Jul 06 20:09:31.350020 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:31.349993 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-6wbdc_dd268b94-55ca-4307-beb5-29fdc2d18c78/payload-pre-processing/0.log" Jul 06 20:09:31.395208 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:31.395153 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6dcdb5648f-whl8g_677b6ba1-3252-4777-88fb-b4238918b699/router/0.log" Jul 06 20:09:32.097634 ip-10-0-135-23 kubenswrapper[2575]: I0706 20:09:32.097588 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-2k5tq_9f4de39b-3797-4653-99bd-150dd6dd9955/openshift-lws-operator/0.log"