Jul 08 19:03:35.266879 ip-10-0-131-191 systemd[1]: Starting Kubernetes Kubelet... Jul 08 19:03:35.775703 ip-10-0-131-191 kubenswrapper[2558]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 19:03:35.775703 ip-10-0-131-191 kubenswrapper[2558]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 08 19:03:35.775703 ip-10-0-131-191 kubenswrapper[2558]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 19:03:35.775703 ip-10-0-131-191 kubenswrapper[2558]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 08 19:03:35.775703 ip-10-0-131-191 kubenswrapper[2558]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 19:03:35.775703 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.689583 2558 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 08 19:03:35.775703 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696549 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 19:03:35.775703 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696563 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 19:03:35.775703 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696566 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 19:03:35.775703 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696569 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 19:03:35.775703 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696572 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 19:03:35.775703 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696575 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 19:03:35.775703 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696579 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 19:03:35.775703 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696582 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 19:03:35.764289 ip-10-0-131-191 systemd[1]: Started Kubernetes Kubelet. Jul 08 19:03:36.047790 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696585 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 19:03:36.047790 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696588 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 19:03:36.047790 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696592 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 19:03:36.047790 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696594 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 19:03:36.047790 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696597 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 19:03:36.047790 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696615 2558 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 19:03:36.047790 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696618 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 19:03:36.047790 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696620 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 19:03:36.047790 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696623 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 19:03:36.047790 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696626 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 19:03:36.047790 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696629 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 19:03:36.047790 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696631 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 19:03:36.047790 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696636 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 19:03:36.047790 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696640 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 19:03:36.047790 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696643 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 19:03:36.047790 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696653 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 19:03:36.047790 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696656 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 19:03:36.047790 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696659 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 19:03:36.047790 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696662 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 19:03:36.057417 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696665 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 19:03:36.057417 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696667 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 19:03:36.057417 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696670 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 19:03:36.057417 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696673 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 19:03:36.057417 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696676 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 19:03:36.057417 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696680 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 19:03:36.057417 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696683 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 19:03:36.057417 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696686 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 19:03:36.057417 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696689 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 19:03:36.057417 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696691 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 19:03:36.057417 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696694 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 19:03:36.057417 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696696 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 19:03:36.057417 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696700 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 19:03:36.057417 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696703 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 19:03:36.057417 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696708 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 19:03:36.057417 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696712 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 19:03:36.057417 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696715 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 19:03:36.057417 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696718 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 19:03:36.057417 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696721 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 19:03:36.058191 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696724 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 19:03:36.058191 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696727 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 19:03:36.058191 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696729 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 19:03:36.058191 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696732 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 19:03:36.058191 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696735 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 19:03:36.058191 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696738 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 19:03:36.058191 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696740 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 19:03:36.058191 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696743 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 19:03:36.058191 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696746 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 19:03:36.058191 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696748 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 19:03:36.058191 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696751 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 19:03:36.058191 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696754 2558 feature_gate.go:328] unrecognized feature gate: Example Jul 08 19:03:36.058191 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696756 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 19:03:36.058191 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696759 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 19:03:36.058191 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696761 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 19:03:36.058191 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696764 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 19:03:36.058191 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696766 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 19:03:36.058191 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696769 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 19:03:36.058191 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696771 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 19:03:36.058191 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696774 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 19:03:36.058997 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696776 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 19:03:36.058997 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696779 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 19:03:36.058997 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696781 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 19:03:36.058997 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696784 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 19:03:36.058997 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696786 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 19:03:36.058997 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696789 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 19:03:36.058997 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696792 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 19:03:36.058997 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696795 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 19:03:36.058997 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696797 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 19:03:36.058997 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696800 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 19:03:36.058997 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696802 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 19:03:36.058997 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696805 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 19:03:36.058997 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696808 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 19:03:36.058997 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696810 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 19:03:36.058997 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696813 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 19:03:36.058997 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696815 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 19:03:36.058997 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696818 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 19:03:36.058997 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696821 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 19:03:36.058997 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696825 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 19:03:36.058997 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696828 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 19:03:36.059597 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696830 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 19:03:36.059597 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696833 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 19:03:36.059597 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696836 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 19:03:36.059597 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696838 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 19:03:36.059597 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696841 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 19:03:36.059597 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696843 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 19:03:36.059597 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696846 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 19:03:36.059597 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696848 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 19:03:36.059597 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696850 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 19:03:36.059597 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696853 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 19:03:36.059597 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696855 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 19:03:36.059597 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696858 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 19:03:36.059597 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.696860 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 19:03:36.059597 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697547 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 19:03:36.059597 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697553 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 19:03:36.059597 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697556 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 19:03:36.059597 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697558 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 19:03:36.059597 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697561 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 19:03:36.059597 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697564 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 19:03:36.060253 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697567 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 19:03:36.060253 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697569 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 19:03:36.060253 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697572 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 19:03:36.060253 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697575 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 19:03:36.060253 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697577 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 19:03:36.060253 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697580 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 19:03:36.060253 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697583 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 19:03:36.060253 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697586 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 19:03:36.060253 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697588 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 19:03:36.060253 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697591 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 19:03:36.060253 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697594 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 19:03:36.060253 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697597 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 19:03:36.060253 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697618 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 19:03:36.060253 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697621 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 19:03:36.060253 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697624 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 19:03:36.060253 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697627 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 19:03:36.060253 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697629 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 19:03:36.060253 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697632 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 19:03:36.060253 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697634 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 19:03:36.060253 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697637 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 19:03:36.060253 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697640 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 19:03:36.060910 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697642 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 19:03:36.060910 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697646 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 19:03:36.060910 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697648 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 19:03:36.060910 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697651 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 19:03:36.060910 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697653 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 19:03:36.060910 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697656 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 19:03:36.060910 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697659 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 19:03:36.060910 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697661 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 19:03:36.060910 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697664 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 19:03:36.060910 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697667 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 19:03:36.060910 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697670 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 19:03:36.060910 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697673 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 19:03:36.060910 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697675 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 19:03:36.060910 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697678 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 19:03:36.060910 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697680 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 19:03:36.060910 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697683 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 19:03:36.060910 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697685 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 19:03:36.060910 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697688 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 19:03:36.060910 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697691 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 19:03:36.061507 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697693 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 19:03:36.061507 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697696 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 19:03:36.061507 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697698 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 19:03:36.061507 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697701 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 19:03:36.061507 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697704 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 19:03:36.061507 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697707 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 19:03:36.061507 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697710 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 19:03:36.061507 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697713 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 19:03:36.061507 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697716 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 19:03:36.061507 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697719 2558 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 19:03:36.061507 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697721 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 19:03:36.061507 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697724 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 19:03:36.061507 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697726 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 19:03:36.061507 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697729 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 19:03:36.061507 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697732 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 19:03:36.061507 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697735 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 19:03:36.061507 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697738 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 19:03:36.061507 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697741 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 19:03:36.061507 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697744 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 19:03:36.061507 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697747 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 19:03:36.062097 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697752 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 19:03:36.062097 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697755 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 19:03:36.062097 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697757 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 19:03:36.062097 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697761 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 19:03:36.062097 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697763 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 19:03:36.062097 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697766 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 19:03:36.062097 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697769 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 19:03:36.062097 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697773 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 19:03:36.062097 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697775 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 19:03:36.062097 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697778 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 19:03:36.062097 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697781 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 19:03:36.062097 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697783 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 19:03:36.062097 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697786 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 19:03:36.062097 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697788 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 19:03:36.062097 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697791 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 19:03:36.062097 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697793 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 19:03:36.062097 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697796 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 19:03:36.062097 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697799 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 19:03:36.062097 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697802 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 19:03:36.062097 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697804 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 19:03:36.062736 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697807 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 19:03:36.062736 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697810 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 19:03:36.062736 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697812 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 19:03:36.062736 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697815 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 19:03:36.062736 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697819 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 19:03:36.062736 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697823 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 19:03:36.062736 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697827 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 19:03:36.062736 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697831 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 19:03:36.062736 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697835 2558 feature_gate.go:328] unrecognized feature gate: Example Jul 08 19:03:36.062736 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697838 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 19:03:36.062736 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697841 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 19:03:36.062736 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697844 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 19:03:36.062736 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.697848 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 19:03:36.062736 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698226 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 19:03:36.062736 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698231 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 19:03:36.062736 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698234 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 19:03:36.062736 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698238 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 19:03:36.062736 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698240 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 19:03:36.062736 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698243 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 19:03:36.063306 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698247 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 19:03:36.063306 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698249 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 19:03:36.063306 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698252 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 19:03:36.063306 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698255 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 19:03:36.063306 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698258 2558 feature_gate.go:328] unrecognized feature gate: Example Jul 08 19:03:36.063306 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698261 2558 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 19:03:36.063306 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698265 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 19:03:36.063306 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698268 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 19:03:36.063306 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698271 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 19:03:36.063306 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698274 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 19:03:36.063306 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698278 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 19:03:36.063306 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698281 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 19:03:36.063306 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698284 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 19:03:36.063306 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698287 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 19:03:36.063306 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698289 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 19:03:36.063306 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698292 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 19:03:36.063306 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698294 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 19:03:36.063306 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698297 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 19:03:36.063306 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698299 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 19:03:36.063306 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698302 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 19:03:36.064101 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698304 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 19:03:36.064101 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698306 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 19:03:36.064101 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698309 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 19:03:36.064101 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698312 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 19:03:36.064101 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698317 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 19:03:36.064101 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698320 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 19:03:36.064101 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698323 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 19:03:36.064101 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698326 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 19:03:36.064101 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698328 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 19:03:36.064101 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698331 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 19:03:36.064101 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698333 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 19:03:36.064101 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698336 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 19:03:36.064101 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698338 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 19:03:36.064101 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698341 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 19:03:36.064101 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698343 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 19:03:36.064101 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698346 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 19:03:36.064101 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698348 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 19:03:36.064101 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698351 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 19:03:36.064101 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698358 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 19:03:36.064101 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698360 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 19:03:36.064921 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698363 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 19:03:36.064921 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698365 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 19:03:36.064921 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698368 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 19:03:36.064921 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698373 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 19:03:36.064921 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698376 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 19:03:36.064921 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698378 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 19:03:36.064921 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698381 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 19:03:36.064921 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698383 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 19:03:36.064921 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698386 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 19:03:36.064921 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698388 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 19:03:36.064921 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698391 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 19:03:36.064921 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698393 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 19:03:36.064921 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698396 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 19:03:36.064921 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698398 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 19:03:36.064921 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698401 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 19:03:36.064921 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698404 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 19:03:36.064921 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698409 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 19:03:36.064921 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698412 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 19:03:36.064921 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698415 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 19:03:36.065581 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698418 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 19:03:36.065581 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698420 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 19:03:36.065581 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698423 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 19:03:36.065581 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698425 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 19:03:36.065581 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698428 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 19:03:36.065581 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698431 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 19:03:36.065581 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698433 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 19:03:36.065581 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698436 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 19:03:36.065581 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698438 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 19:03:36.065581 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698441 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 19:03:36.065581 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698443 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 19:03:36.065581 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698447 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 19:03:36.065581 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698451 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 19:03:36.065581 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698454 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 19:03:36.065581 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698458 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 19:03:36.065581 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698460 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 19:03:36.065581 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698463 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 19:03:36.065581 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698466 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 19:03:36.065581 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698469 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 19:03:36.066110 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698471 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 19:03:36.066110 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698473 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 19:03:36.066110 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698476 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 19:03:36.066110 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698479 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 19:03:36.066110 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698481 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 19:03:36.066110 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698484 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 19:03:36.066110 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698486 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 19:03:36.066110 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698489 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 19:03:36.066110 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698491 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 19:03:36.066110 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698494 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 19:03:36.066110 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698498 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 19:03:36.066110 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698501 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 19:03:36.066110 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698504 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 19:03:36.066110 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698506 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 19:03:36.066110 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.698509 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 19:03:36.066110 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698580 2558 flags.go:64] FLAG: --address="0.0.0.0" Jul 08 19:03:36.066110 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698591 2558 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 08 19:03:36.066110 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698598 2558 flags.go:64] FLAG: --anonymous-auth="true" Jul 08 19:03:36.066110 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698615 2558 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 08 19:03:36.066110 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698619 2558 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 08 19:03:36.066110 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698622 2558 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 08 19:03:36.066721 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698626 2558 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 08 19:03:36.066721 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698630 2558 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 08 19:03:36.066721 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698634 2558 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 08 19:03:36.066721 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698637 2558 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 08 19:03:36.066721 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698640 2558 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 08 19:03:36.066721 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698643 2558 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 08 19:03:36.066721 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698646 2558 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 08 19:03:36.066721 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698649 2558 flags.go:64] FLAG: --cgroup-root="" Jul 08 19:03:36.066721 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698652 2558 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 08 19:03:36.066721 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698655 2558 flags.go:64] FLAG: --client-ca-file="" Jul 08 19:03:36.066721 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698659 2558 flags.go:64] FLAG: --cloud-provider="external" Jul 08 19:03:36.066721 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698662 2558 flags.go:64] FLAG: --cluster-dns="[]" Jul 08 19:03:36.066721 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698669 2558 flags.go:64] FLAG: --cluster-domain="" Jul 08 19:03:36.066721 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698672 2558 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 08 19:03:36.066721 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698675 2558 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 08 19:03:36.066721 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698678 2558 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 08 19:03:36.066721 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698681 2558 flags.go:64] FLAG: --container-log-max-files="5" Jul 08 19:03:36.066721 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698685 2558 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 08 19:03:36.066721 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698688 2558 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 08 19:03:36.066721 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698691 2558 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 08 19:03:36.066721 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698694 2558 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 08 19:03:36.066721 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698699 2558 flags.go:64] FLAG: --contention-profiling="false" Jul 08 19:03:36.066721 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698703 2558 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 08 19:03:36.066721 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698706 2558 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 08 19:03:36.067341 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698709 2558 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 08 19:03:36.067341 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698711 2558 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 08 19:03:36.067341 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698715 2558 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 08 19:03:36.067341 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698718 2558 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 08 19:03:36.067341 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698722 2558 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 08 19:03:36.067341 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698724 2558 flags.go:64] FLAG: --enable-load-reader="false" Jul 08 19:03:36.067341 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698727 2558 flags.go:64] FLAG: --enable-server="true" Jul 08 19:03:36.067341 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698730 2558 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 08 19:03:36.067341 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698735 2558 flags.go:64] FLAG: --event-burst="100" Jul 08 19:03:36.067341 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698738 2558 flags.go:64] FLAG: --event-qps="50" Jul 08 19:03:36.067341 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698740 2558 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 08 19:03:36.067341 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698743 2558 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 08 19:03:36.067341 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698746 2558 flags.go:64] FLAG: --eviction-hard="" Jul 08 19:03:36.067341 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698750 2558 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 08 19:03:36.067341 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698752 2558 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 08 19:03:36.067341 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698755 2558 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 08 19:03:36.067341 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698758 2558 flags.go:64] FLAG: --eviction-soft="" Jul 08 19:03:36.067341 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698761 2558 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 08 19:03:36.067341 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698764 2558 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 08 19:03:36.067341 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698767 2558 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 08 19:03:36.067341 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698770 2558 flags.go:64] FLAG: --experimental-mounter-path="" Jul 08 19:03:36.067341 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698773 2558 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 08 19:03:36.067341 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698776 2558 flags.go:64] FLAG: --fail-swap-on="true" Jul 08 19:03:36.067341 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698778 2558 flags.go:64] FLAG: --feature-gates="" Jul 08 19:03:36.067341 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698782 2558 flags.go:64] FLAG: --file-check-frequency="20s" Jul 08 19:03:36.068097 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698785 2558 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 08 19:03:36.068097 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698788 2558 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 08 19:03:36.068097 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698790 2558 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 08 19:03:36.068097 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698793 2558 flags.go:64] FLAG: --healthz-port="10248" Jul 08 19:03:36.068097 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698798 2558 flags.go:64] FLAG: --help="false" Jul 08 19:03:36.068097 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698802 2558 flags.go:64] FLAG: --hostname-override="ip-10-0-131-191.ec2.internal" Jul 08 19:03:36.068097 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698805 2558 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 08 19:03:36.068097 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698808 2558 flags.go:64] FLAG: --http-check-frequency="20s" Jul 08 19:03:36.068097 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698811 2558 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 08 19:03:36.068097 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698814 2558 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 08 19:03:36.068097 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698818 2558 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 08 19:03:36.068097 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698820 2558 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 08 19:03:36.068097 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698823 2558 flags.go:64] FLAG: --image-service-endpoint="" Jul 08 19:03:36.068097 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698826 2558 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 08 19:03:36.068097 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698829 2558 flags.go:64] FLAG: --kube-api-burst="100" Jul 08 19:03:36.068097 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698832 2558 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 08 19:03:36.068097 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698835 2558 flags.go:64] FLAG: --kube-api-qps="50" Jul 08 19:03:36.068097 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698838 2558 flags.go:64] FLAG: --kube-reserved="" Jul 08 19:03:36.068097 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698840 2558 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 08 19:03:36.068097 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698843 2558 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 08 19:03:36.068097 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698846 2558 flags.go:64] FLAG: --kubelet-cgroups="" Jul 08 19:03:36.068097 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698849 2558 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 08 19:03:36.068097 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698852 2558 flags.go:64] FLAG: --lock-file="" Jul 08 19:03:36.068097 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698855 2558 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 08 19:03:36.068744 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698857 2558 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 08 19:03:36.068744 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698860 2558 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 08 19:03:36.068744 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698865 2558 flags.go:64] FLAG: --log-json-split-stream="false" Jul 08 19:03:36.068744 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698869 2558 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 08 19:03:36.068744 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698872 2558 flags.go:64] FLAG: --log-text-split-stream="false" Jul 08 19:03:36.068744 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698874 2558 flags.go:64] FLAG: --logging-format="text" Jul 08 19:03:36.068744 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698877 2558 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 08 19:03:36.068744 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698880 2558 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 08 19:03:36.068744 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698883 2558 flags.go:64] FLAG: --manifest-url="" Jul 08 19:03:36.068744 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698886 2558 flags.go:64] FLAG: --manifest-url-header="" Jul 08 19:03:36.068744 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698890 2558 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 08 19:03:36.068744 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698893 2558 flags.go:64] FLAG: --max-open-files="1000000" Jul 08 19:03:36.068744 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698897 2558 flags.go:64] FLAG: --max-pods="110" Jul 08 19:03:36.068744 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698902 2558 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 08 19:03:36.068744 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698905 2558 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 08 19:03:36.068744 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698907 2558 flags.go:64] FLAG: --memory-manager-policy="None" Jul 08 19:03:36.068744 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698910 2558 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 08 19:03:36.068744 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698913 2558 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 08 19:03:36.068744 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698916 2558 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 08 19:03:36.068744 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698919 2558 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 08 19:03:36.068744 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698926 2558 flags.go:64] FLAG: --node-status-max-images="50" Jul 08 19:03:36.068744 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698929 2558 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 08 19:03:36.068744 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698932 2558 flags.go:64] FLAG: --oom-score-adj="-999" Jul 08 19:03:36.068744 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698935 2558 flags.go:64] FLAG: --pod-cidr="" Jul 08 19:03:36.069381 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698938 2558 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 08 19:03:36.069381 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698943 2558 flags.go:64] FLAG: --pod-manifest-path="" Jul 08 19:03:36.069381 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698946 2558 flags.go:64] FLAG: --pod-max-pids="-1" Jul 08 19:03:36.069381 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698949 2558 flags.go:64] FLAG: --pods-per-core="0" Jul 08 19:03:36.069381 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698951 2558 flags.go:64] FLAG: --port="10250" Jul 08 19:03:36.069381 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698954 2558 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 08 19:03:36.069381 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698957 2558 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09187cd0d657c6bc7" Jul 08 19:03:36.069381 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698960 2558 flags.go:64] FLAG: --qos-reserved="" Jul 08 19:03:36.069381 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698963 2558 flags.go:64] FLAG: --read-only-port="10255" Jul 08 19:03:36.069381 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698966 2558 flags.go:64] FLAG: --register-node="true" Jul 08 19:03:36.069381 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698968 2558 flags.go:64] FLAG: --register-with-taints="" Jul 08 19:03:36.069381 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698972 2558 flags.go:64] FLAG: --registry-burst="10" Jul 08 19:03:36.069381 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698976 2558 flags.go:64] FLAG: --registry-qps="5" Jul 08 19:03:36.069381 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698979 2558 flags.go:64] FLAG: --reserved-cpus="" Jul 08 19:03:36.069381 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698981 2558 flags.go:64] FLAG: --reserved-memory="" Jul 08 19:03:36.069381 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698985 2558 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 08 19:03:36.069381 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698988 2558 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 08 19:03:36.069381 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698991 2558 flags.go:64] FLAG: --rotate-certificates="false" Jul 08 19:03:36.069381 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698993 2558 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 08 19:03:36.069381 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.698996 2558 flags.go:64] FLAG: --runonce="false" Jul 08 19:03:36.069381 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.699002 2558 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 08 19:03:36.069381 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.699005 2558 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 08 19:03:36.069381 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.699009 2558 flags.go:64] FLAG: --seccomp-default="false" Jul 08 19:03:36.069381 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.699012 2558 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 08 19:03:36.069381 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.699015 2558 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 08 19:03:36.070386 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.699018 2558 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 08 19:03:36.070386 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.699021 2558 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 08 19:03:36.070386 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.699024 2558 flags.go:64] FLAG: --storage-driver-password="root" Jul 08 19:03:36.070386 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.699027 2558 flags.go:64] FLAG: --storage-driver-secure="false" Jul 08 19:03:36.070386 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.699029 2558 flags.go:64] FLAG: --storage-driver-table="stats" Jul 08 19:03:36.070386 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.699032 2558 flags.go:64] FLAG: --storage-driver-user="root" Jul 08 19:03:36.070386 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.699035 2558 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 08 19:03:36.070386 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.699038 2558 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 08 19:03:36.070386 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.699040 2558 flags.go:64] FLAG: --system-cgroups="" Jul 08 19:03:36.070386 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.699043 2558 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 08 19:03:36.070386 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.699048 2558 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 08 19:03:36.070386 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.699051 2558 flags.go:64] FLAG: --tls-cert-file="" Jul 08 19:03:36.070386 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.699054 2558 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 08 19:03:36.070386 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.699060 2558 flags.go:64] FLAG: --tls-min-version="" Jul 08 19:03:36.070386 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.699063 2558 flags.go:64] FLAG: --tls-private-key-file="" Jul 08 19:03:36.070386 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.699066 2558 flags.go:64] FLAG: --topology-manager-policy="none" Jul 08 19:03:36.070386 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.699069 2558 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 08 19:03:36.070386 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.699071 2558 flags.go:64] FLAG: --topology-manager-scope="container" Jul 08 19:03:36.070386 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.699074 2558 flags.go:64] FLAG: --v="2" Jul 08 19:03:36.070386 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.699078 2558 flags.go:64] FLAG: --version="false" Jul 08 19:03:36.070386 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.699086 2558 flags.go:64] FLAG: --vmodule="" Jul 08 19:03:36.070386 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.699090 2558 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 08 19:03:36.070386 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.699093 2558 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 08 19:03:36.070386 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699202 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 19:03:36.071033 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699207 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 19:03:36.071033 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699211 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 19:03:36.071033 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699214 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 19:03:36.071033 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699218 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 19:03:36.071033 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699221 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 19:03:36.071033 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699225 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 19:03:36.071033 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699228 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 19:03:36.071033 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699231 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 19:03:36.071033 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699234 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 19:03:36.071033 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699236 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 19:03:36.071033 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699239 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 19:03:36.071033 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699241 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 19:03:36.071033 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699244 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 19:03:36.071033 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699247 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 19:03:36.071033 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699249 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 19:03:36.071033 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699252 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 19:03:36.071033 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699254 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 19:03:36.071033 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699257 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 19:03:36.071033 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699259 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 19:03:36.071033 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699262 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 19:03:36.246026 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699264 2558 feature_gate.go:328] unrecognized feature gate: Example Jul 08 19:03:36.246026 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699267 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 19:03:36.246026 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699269 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 19:03:36.246026 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699272 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 19:03:36.246026 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699274 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 19:03:36.246026 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699277 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 19:03:36.246026 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699279 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 19:03:36.246026 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699282 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 19:03:36.246026 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699284 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 19:03:36.246026 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699287 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 19:03:36.246026 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699290 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 19:03:36.246026 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699292 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 19:03:36.246026 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699295 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 19:03:36.246026 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699297 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 19:03:36.246026 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699300 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 19:03:36.246026 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699303 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 19:03:36.246026 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699306 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 19:03:36.246026 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699311 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 19:03:36.246026 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699313 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 19:03:36.246026 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699316 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 19:03:36.246838 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699319 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 19:03:36.246838 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699321 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 19:03:36.246838 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699324 2558 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 19:03:36.246838 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699326 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 19:03:36.246838 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699329 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 19:03:36.246838 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699332 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 19:03:36.246838 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699335 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 19:03:36.246838 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699337 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 19:03:36.246838 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699340 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 19:03:36.246838 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699343 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 19:03:36.246838 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699347 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 19:03:36.246838 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699349 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 19:03:36.246838 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699352 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 19:03:36.246838 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699354 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 19:03:36.246838 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699357 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 19:03:36.246838 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699360 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 19:03:36.246838 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699362 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 19:03:36.246838 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699364 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 19:03:36.246838 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699367 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 19:03:36.246838 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699370 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 19:03:36.247460 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699372 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 19:03:36.247460 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699375 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 19:03:36.247460 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699377 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 19:03:36.247460 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699380 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 19:03:36.247460 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699383 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 19:03:36.247460 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699385 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 19:03:36.247460 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699389 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 19:03:36.247460 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699394 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 19:03:36.247460 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699397 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 19:03:36.247460 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699401 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 19:03:36.247460 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699404 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 19:03:36.247460 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699407 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 19:03:36.247460 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699409 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 19:03:36.247460 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699412 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 19:03:36.247460 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699415 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 19:03:36.247460 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699417 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 19:03:36.247460 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699420 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 19:03:36.247460 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699423 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 19:03:36.247460 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699425 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 19:03:36.248083 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699428 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 19:03:36.248083 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699431 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 19:03:36.248083 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699433 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 19:03:36.248083 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699436 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 19:03:36.248083 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699439 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 19:03:36.248083 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699442 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 19:03:36.248083 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699444 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 19:03:36.248083 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699447 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 19:03:36.248083 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699449 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 19:03:36.248083 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699451 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 19:03:36.248083 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699454 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 19:03:36.248083 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699457 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 19:03:36.248083 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699459 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 19:03:36.248083 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699462 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 19:03:36.248083 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699464 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 19:03:36.248083 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699467 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 19:03:36.248083 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699470 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 19:03:36.248083 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699473 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 19:03:36.248083 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.699475 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 19:03:36.248927 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.700199 2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 19:03:36.248927 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.707060 2558 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 08 19:03:36.248927 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.707077 2558 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 08 19:03:36.248927 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.707946 2558 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 08 19:03:36.248927 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.707954 2558 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 08 19:03:36.248927 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708019 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 19:03:36.248927 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708024 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 19:03:36.248927 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708027 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 19:03:36.248927 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708030 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 19:03:36.248927 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708034 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 19:03:36.248927 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708038 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 19:03:36.248927 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708044 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 19:03:36.248927 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708049 2558 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 19:03:36.248927 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708053 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 19:03:36.248927 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708056 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 19:03:36.249546 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708058 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 19:03:36.249546 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708061 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 19:03:36.249546 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708064 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 19:03:36.249546 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708066 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 19:03:36.249546 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708069 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 19:03:36.249546 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708072 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 19:03:36.249546 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708074 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 19:03:36.249546 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708077 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 19:03:36.249546 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708080 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 19:03:36.249546 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708082 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 19:03:36.249546 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708086 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 19:03:36.249546 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708088 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 19:03:36.249546 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708092 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 19:03:36.249546 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708096 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 19:03:36.249546 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708099 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 19:03:36.249546 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708102 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 19:03:36.249546 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708106 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 19:03:36.249546 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708109 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 19:03:36.249546 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708111 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 19:03:36.250166 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708120 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 19:03:36.250166 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708123 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 19:03:36.250166 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708126 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 19:03:36.250166 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708128 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 19:03:36.250166 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708131 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 19:03:36.250166 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708133 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 19:03:36.250166 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708136 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 19:03:36.250166 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708139 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 19:03:36.250166 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708141 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 19:03:36.250166 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708144 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 19:03:36.250166 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708146 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 19:03:36.250166 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708149 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 19:03:36.250166 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708151 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 19:03:36.250166 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708154 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 19:03:36.250166 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708157 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 19:03:36.250166 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708159 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 19:03:36.250166 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708162 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 19:03:36.250166 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708164 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 19:03:36.250166 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708167 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 19:03:36.250166 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708169 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 19:03:36.250709 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708171 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 19:03:36.250709 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708174 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 19:03:36.250709 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708177 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 19:03:36.250709 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708181 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 19:03:36.250709 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708184 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 19:03:36.250709 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708186 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 19:03:36.250709 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708189 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 19:03:36.250709 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708191 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 19:03:36.250709 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708194 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 19:03:36.250709 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708197 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 19:03:36.250709 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708200 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 19:03:36.250709 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708202 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 19:03:36.250709 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708205 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 19:03:36.250709 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708213 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 19:03:36.250709 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708215 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 19:03:36.250709 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708218 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 19:03:36.250709 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708220 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 19:03:36.250709 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708223 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 19:03:36.250709 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708226 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 19:03:36.250709 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708228 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 19:03:36.251256 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708231 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 19:03:36.251256 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708233 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 19:03:36.251256 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708236 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 19:03:36.251256 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708238 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 19:03:36.251256 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708241 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 19:03:36.251256 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708244 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 19:03:36.251256 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708246 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 19:03:36.251256 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708251 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 19:03:36.251256 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708255 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 19:03:36.251256 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708259 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 19:03:36.251256 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708262 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 19:03:36.251256 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708264 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 19:03:36.251256 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708267 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 19:03:36.251256 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708269 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 19:03:36.251256 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708272 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 19:03:36.251256 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708276 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 19:03:36.251256 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708279 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 19:03:36.251256 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708281 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 19:03:36.251256 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708284 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 19:03:36.251818 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708288 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 19:03:36.251818 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708292 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 19:03:36.251818 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708296 2558 feature_gate.go:328] unrecognized feature gate: Example Jul 08 19:03:36.251818 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708299 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 19:03:36.251818 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708301 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 19:03:36.251818 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708304 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 19:03:36.251818 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708306 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 19:03:36.251818 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708315 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 19:03:36.251818 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708318 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 19:03:36.251818 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708320 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 19:03:36.251818 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708323 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 19:03:36.251818 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.708328 2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 19:03:36.251818 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708492 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 19:03:36.251818 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708498 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 19:03:36.252192 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708501 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 19:03:36.252192 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708504 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 19:03:36.252192 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708507 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 19:03:36.252192 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708510 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 19:03:36.252192 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708512 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 19:03:36.252192 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708515 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 19:03:36.252192 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708518 2558 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 19:03:36.252192 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708521 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 19:03:36.252192 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708523 2558 feature_gate.go:328] unrecognized feature gate: Example Jul 08 19:03:36.252192 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708526 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 19:03:36.252192 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708528 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 19:03:36.252192 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708531 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 19:03:36.252192 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708533 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 19:03:36.252192 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708536 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 19:03:36.252192 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708540 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 19:03:36.252192 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708542 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 19:03:36.252192 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708545 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 19:03:36.252192 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708547 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 19:03:36.252192 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708550 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 19:03:36.252192 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708552 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 19:03:36.252773 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708555 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 19:03:36.252773 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708558 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 19:03:36.252773 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708561 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 19:03:36.252773 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708563 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 19:03:36.252773 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708566 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 19:03:36.252773 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708568 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 19:03:36.252773 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708577 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 19:03:36.252773 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708580 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 19:03:36.252773 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708582 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 19:03:36.252773 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708585 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 19:03:36.252773 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708587 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 19:03:36.252773 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708590 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 19:03:36.252773 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708592 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 19:03:36.252773 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708594 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 19:03:36.252773 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708597 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 19:03:36.252773 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708614 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 19:03:36.252773 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708617 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 19:03:36.252773 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708620 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 19:03:36.252773 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708622 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 19:03:36.252773 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708625 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 19:03:36.252773 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708627 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 19:03:36.253329 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708630 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 19:03:36.253329 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708634 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 19:03:36.253329 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708637 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 19:03:36.253329 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708641 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 19:03:36.253329 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708645 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 19:03:36.253329 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708650 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 19:03:36.253329 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708653 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 19:03:36.253329 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708656 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 19:03:36.253329 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708658 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 19:03:36.253329 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708661 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 19:03:36.253329 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708664 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 19:03:36.253329 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708667 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 19:03:36.253329 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708673 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 19:03:36.253329 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708678 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 19:03:36.253329 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708682 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 19:03:36.253329 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708684 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 19:03:36.253329 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708687 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 19:03:36.253329 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708690 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 19:03:36.253329 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708700 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 19:03:36.253887 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708702 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 19:03:36.253887 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708705 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 19:03:36.253887 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708707 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 19:03:36.253887 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708710 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 19:03:36.253887 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708712 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 19:03:36.253887 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708715 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 19:03:36.253887 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708717 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 19:03:36.253887 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708720 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 19:03:36.253887 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708722 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 19:03:36.253887 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708725 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 19:03:36.253887 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708727 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 19:03:36.253887 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708731 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 19:03:36.253887 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708734 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 19:03:36.253887 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708737 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 19:03:36.253887 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708740 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 19:03:36.253887 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708743 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 19:03:36.253887 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708745 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 19:03:36.253887 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708748 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 19:03:36.253887 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708752 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 19:03:36.254391 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708755 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 19:03:36.254391 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708757 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 19:03:36.254391 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708760 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 19:03:36.254391 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708762 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 19:03:36.254391 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708764 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 19:03:36.254391 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708767 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 19:03:36.254391 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708770 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 19:03:36.254391 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708773 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 19:03:36.254391 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708776 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 19:03:36.254391 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708779 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 19:03:36.254391 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708781 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 19:03:36.254391 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708784 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 19:03:36.254391 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708786 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 19:03:36.254391 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708794 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 19:03:36.254391 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708797 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 19:03:36.254391 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708800 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 19:03:36.254391 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708802 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 19:03:36.254391 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:35.708805 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 19:03:36.254946 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.708809 2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 19:03:36.254946 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.708962 2558 server.go:962] "Client rotation is on, will bootstrap in background" Jul 08 19:03:36.254946 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.711065 2558 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 08 19:03:36.254946 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.712708 2558 server.go:1019] "Starting client certificate rotation" Jul 08 19:03:36.254946 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.712839 2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 08 19:03:36.254946 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.712894 2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 08 19:03:36.254946 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.715691 2558 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 08 19:03:36.254946 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.719273 2558 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 08 19:03:36.254946 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.727903 2558 log.go:25] "Validated CRI v1 runtime API" Jul 08 19:03:36.254946 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.730186 2558 log.go:25] "Validated CRI v1 image API" Jul 08 19:03:36.254946 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.730955 2558 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 08 19:03:36.254946 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.734066 2558 fs.go:135] Filesystem UUIDs: map[3fefdfd7-f05a-4ee1-bfac-3abec63d907d:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 909c1d57-20a9-4102-ac8e-bc0a3847d72f:/dev/nvme0n1p3] Jul 08 19:03:36.255307 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.734079 2558 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 08 19:03:36.255404 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.739289 2558 manager.go:216] Machine: {Timestamp:2026-07-08 19:03:35.737939722 +0000 UTC m=+0.348867243 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098979 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c5719360fecb231a1ae141dd0e049 SystemUUID:ec2c5719-360f-ecb2-31a1-ae141dd0e049 BootID:7491d4df-b8d8-4a7e-b7a7-cf1bc21b84cd Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:06:d0:99:04:69 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:06:d0:99:04:69 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:e1:7c:69:6d:ce Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 08 19:03:36.255404 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.739386 2558 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 08 19:03:36.255404 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.739455 2558 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 08 19:03:36.255404 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.743011 2558 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 08 19:03:36.255404 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.743036 2558 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-191.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 08 19:03:36.255404 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.743786 2558 topology_manager.go:138] "Creating topology manager with none policy" Jul 08 19:03:36.255404 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.743795 2558 container_manager_linux.go:309] "Creating device plugin manager" Jul 08 19:03:36.255404 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.743802 2558 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 08 19:03:36.255711 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.743999 2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 08 19:03:36.255711 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.744590 2558 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 08 19:03:36.255711 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.745914 2558 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 08 19:03:36.255711 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.748891 2558 state_mem.go:36] "Initialized new in-memory state store" Jul 08 19:03:36.255711 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.749002 2558 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 08 19:03:36.255711 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.749028 2558 kubelet.go:486] "Attempting to sync node with API server" Jul 08 19:03:36.255711 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.749040 2558 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 08 19:03:36.255711 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.749054 2558 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 08 19:03:36.255711 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.749065 2558 kubelet.go:398] "Adding apiserver pod source" Jul 08 19:03:36.255711 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.749074 2558 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 08 19:03:36.255711 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.749443 2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 08 19:03:36.255711 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.749459 2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 08 19:03:36.255711 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.752198 2558 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 08 19:03:36.255711 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.754347 2558 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 08 19:03:36.255711 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.754396 2558 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 08 19:03:36.255711 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.756133 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 08 19:03:36.255711 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.756148 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 08 19:03:36.255711 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.756154 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 08 19:03:36.255711 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.756159 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 08 19:03:36.255711 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.756168 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 08 19:03:36.255711 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.756174 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 08 19:03:36.255711 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.756182 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 08 19:03:36.255711 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.756186 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 08 19:03:36.255711 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.756193 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 08 19:03:36.255711 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.756198 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 08 19:03:36.255711 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.756211 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 08 19:03:36.255711 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.756216 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 08 19:03:36.255711 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.756970 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 08 19:03:36.255711 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.756976 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 08 19:03:36.255711 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.760199 2558 server.go:1268] "Started kubelet" Jul 08 19:03:36.255711 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.763660 2558 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-191.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 08 19:03:36.255711 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.763784 2558 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 08 19:03:36.255711 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.763841 2558 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 08 19:03:36.255711 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.760343 2558 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 08 19:03:36.255711 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.765820 2558 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 08 19:03:36.255711 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:35.766724 2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-191.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 08 19:03:36.255711 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:35.766839 2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 08 19:03:36.255711 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.766997 2558 server.go:310] "Adding debug handlers to kubelet server" Jul 08 19:03:36.255711 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:35.769273 2558 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 08 19:03:36.256769 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:35.772076 2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-191.ec2.internal.18c065d47e006793 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-191.ec2.internal,UID:ip-10-0-131-191.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-191.ec2.internal,},FirstTimestamp:2026-07-08 19:03:35.760144275 +0000 UTC m=+0.371071796,LastTimestamp:2026-07-08 19:03:35.760144275 +0000 UTC m=+0.371071796,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-191.ec2.internal,}" Jul 08 19:03:36.256769 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.775121 2558 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 08 19:03:36.256769 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.775128 2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 08 19:03:36.256769 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:35.775208 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-191.ec2.internal\" not found" Jul 08 19:03:36.256769 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.775744 2558 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 08 19:03:36.256769 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.775750 2558 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 08 19:03:36.256769 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.775771 2558 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 08 19:03:36.256769 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.775900 2558 reconstruct.go:97] "Volume reconstruction finished" Jul 08 19:03:36.256769 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.775908 2558 reconciler.go:29] "Reconciler: start to sync state" Jul 08 19:03:36.256769 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.776015 2558 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 08 19:03:36.256769 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.776028 2558 factory.go:55] Registering systemd factory Jul 08 19:03:36.256769 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.776037 2558 factory.go:223] Registration of the systemd container factory successfully Jul 08 19:03:36.256769 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.776230 2558 factory.go:153] Registering CRI-O factory Jul 08 19:03:36.256769 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.776243 2558 factory.go:223] Registration of the crio container factory successfully Jul 08 19:03:36.256769 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.776268 2558 factory.go:103] Registering Raw factory Jul 08 19:03:36.256769 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.776280 2558 manager.go:1196] Started watching for new ooms in manager Jul 08 19:03:36.256769 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.776729 2558 manager.go:349] Starting recovery of all containers Jul 08 19:03:36.256769 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:35.777355 2558 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-191.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 08 19:03:36.256769 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:35.777663 2558 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 08 19:03:36.256769 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.785815 2558 manager.go:354] Recovery completed Jul 08 19:03:36.256769 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.790629 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 19:03:36.256769 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.792221 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-191.ec2.internal" event="NodeHasSufficientMemory" Jul 08 19:03:36.256769 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.792259 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-191.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 19:03:36.256769 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.792270 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-191.ec2.internal" event="NodeHasSufficientPID" Jul 08 19:03:36.256769 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.792776 2558 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 08 19:03:36.256769 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.792786 2558 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 08 19:03:36.256769 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.792788 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 19:03:36.256769 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.792799 2558 state_mem.go:36] "Initialized new in-memory state store" Jul 08 19:03:36.256769 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.793391 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-191.ec2.internal" event="NodeHasSufficientMemory" Jul 08 19:03:36.256769 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.793411 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-191.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 19:03:36.256769 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.793425 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-191.ec2.internal" event="NodeHasSufficientPID" Jul 08 19:03:36.256769 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.794197 2558 policy_none.go:49] "None policy: Start" Jul 08 19:03:36.256769 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.794230 2558 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 08 19:03:36.256769 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.794241 2558 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 08 19:03:36.257928 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.795285 2558 policy_none.go:47] "Start" Jul 08 19:03:36.257928 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:35.795293 2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-191.ec2.internal.18c065d47fea3550 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-191.ec2.internal,UID:ip-10-0-131-191.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-191.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-191.ec2.internal,},FirstTimestamp:2026-07-08 19:03:35.792244048 +0000 UTC m=+0.403171570,LastTimestamp:2026-07-08 19:03:35.792244048 +0000 UTC m=+0.403171570,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-191.ec2.internal,}" Jul 08 19:03:36.257928 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:35.816619 2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-191.ec2.internal.18c065d47fea869f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-191.ec2.internal,UID:ip-10-0-131-191.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-191.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-191.ec2.internal,},FirstTimestamp:2026-07-08 19:03:35.792264863 +0000 UTC m=+0.403192384,LastTimestamp:2026-07-08 19:03:35.792264863 +0000 UTC m=+0.403192384,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-191.ec2.internal,}" Jul 08 19:03:36.257928 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:35.831385 2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-191.ec2.internal.18c065d47feaac9a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-191.ec2.internal,UID:ip-10-0-131-191.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-191.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-191.ec2.internal,},FirstTimestamp:2026-07-08 19:03:35.792274586 +0000 UTC m=+0.403202108,LastTimestamp:2026-07-08 19:03:35.792274586 +0000 UTC m=+0.403202108,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-191.ec2.internal,}" Jul 08 19:03:36.257928 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:35.843454 2558 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-191.ec2.internal.18c065d47fea3550\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-191.ec2.internal.18c065d47fea3550 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-191.ec2.internal,UID:ip-10-0-131-191.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-191.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-191.ec2.internal,},FirstTimestamp:2026-07-08 19:03:35.792244048 +0000 UTC m=+0.403171570,LastTimestamp:2026-07-08 19:03:35.793403469 +0000 UTC m=+0.404330989,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-191.ec2.internal,}" Jul 08 19:03:36.257928 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.853004 2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ngnbv" Jul 08 19:03:36.258183 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:35.854597 2558 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-191.ec2.internal.18c065d47fea869f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-191.ec2.internal.18c065d47fea869f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-191.ec2.internal,UID:ip-10-0-131-191.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-191.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-191.ec2.internal,},FirstTimestamp:2026-07-08 19:03:35.792264863 +0000 UTC m=+0.403192384,LastTimestamp:2026-07-08 19:03:35.793417546 +0000 UTC m=+0.404345070,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-191.ec2.internal,}" Jul 08 19:03:36.258183 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.861922 2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ngnbv" Jul 08 19:03:36.258183 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:35.875570 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-191.ec2.internal\" not found" Jul 08 19:03:36.258183 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.891317 2558 manager.go:337] "Starting Device Plugin manager" Jul 08 19:03:36.258183 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:35.891342 2558 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 08 19:03:36.258183 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.891351 2558 server.go:87] "Starting device plugin registration server" Jul 08 19:03:36.258183 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.891562 2558 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 08 19:03:36.258183 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.891573 2558 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 08 19:03:36.258183 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.891670 2558 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 08 19:03:36.258183 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.891761 2558 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 08 19:03:36.258183 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.891770 2558 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 08 19:03:36.258183 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:35.892283 2558 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 08 19:03:36.258183 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:35.892328 2558 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-191.ec2.internal\" not found" Jul 08 19:03:36.258183 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.922043 2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 08 19:03:36.258183 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.923218 2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 08 19:03:36.258183 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.923251 2558 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 08 19:03:36.258183 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.923268 2558 kubelet.go:2448] "Starting kubelet main sync loop" Jul 08 19:03:36.258183 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:35.923297 2558 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 08 19:03:36.258183 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.925865 2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 19:03:36.258183 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:35.988009 2558 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-131-191.ec2.internal\" not found" node="ip-10-0-131-191.ec2.internal" Jul 08 19:03:36.258183 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.992131 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 19:03:36.258183 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.993030 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-191.ec2.internal" event="NodeHasSufficientMemory" Jul 08 19:03:36.258183 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.993059 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-191.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 19:03:36.258183 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.993069 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-191.ec2.internal" event="NodeHasSufficientPID" Jul 08 19:03:36.258183 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:35.993090 2558 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-191.ec2.internal" Jul 08 19:03:36.258183 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:36.003654 2558 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-191.ec2.internal" Jul 08 19:03:36.258183 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:36.003676 2558 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-191.ec2.internal\": node \"ip-10-0-131-191.ec2.internal\" not found" Jul 08 19:03:36.258183 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:36.023666 2558 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-191.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-191.ec2.internal"] Jul 08 19:03:36.258183 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:36.023719 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 19:03:36.258183 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:36.025925 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-191.ec2.internal" event="NodeHasSufficientMemory" Jul 08 19:03:36.448739 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:36.025956 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-191.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 19:03:36.448739 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:36.025969 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-191.ec2.internal" event="NodeHasSufficientPID" Jul 08 19:03:36.448739 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:36.027309 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 19:03:36.448739 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:36.027461 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-191.ec2.internal" Jul 08 19:03:36.448739 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:36.027492 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 19:03:36.448739 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:36.028053 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-191.ec2.internal" event="NodeHasSufficientMemory" Jul 08 19:03:36.448739 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:36.028068 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-191.ec2.internal" event="NodeHasSufficientMemory" Jul 08 19:03:36.448739 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:36.028077 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-191.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 19:03:36.448739 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:36.028088 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-191.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 19:03:36.448739 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:36.028091 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-191.ec2.internal" event="NodeHasSufficientPID" Jul 08 19:03:36.448739 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:36.028098 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-191.ec2.internal" event="NodeHasSufficientPID" Jul 08 19:03:36.448739 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:36.029417 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-191.ec2.internal" Jul 08 19:03:36.448739 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:36.029441 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 19:03:36.448739 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:36.030117 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-191.ec2.internal" event="NodeHasSufficientMemory" Jul 08 19:03:36.448739 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:36.030146 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-191.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 19:03:36.448739 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:36.030161 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-191.ec2.internal" event="NodeHasSufficientPID" Jul 08 19:03:36.448739 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:36.065282 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-191.ec2.internal\" not found" Jul 08 19:03:36.448739 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:36.077475 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/330f763c6aa8f4d35b771e6847835d74-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-191.ec2.internal\" (UID: \"330f763c6aa8f4d35b771e6847835d74\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-191.ec2.internal" Jul 08 19:03:36.448739 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:36.077496 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ce1e796271b5d048919b6c245c7db2fc-config\") pod \"kube-apiserver-proxy-ip-10-0-131-191.ec2.internal\" (UID: \"ce1e796271b5d048919b6c245c7db2fc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-191.ec2.internal" Jul 08 19:03:36.448739 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:36.077512 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/330f763c6aa8f4d35b771e6847835d74-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-191.ec2.internal\" (UID: \"330f763c6aa8f4d35b771e6847835d74\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-191.ec2.internal" Jul 08 19:03:36.448739 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:36.077582 2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-191.ec2.internal\" not found" node="ip-10-0-131-191.ec2.internal" Jul 08 19:03:36.448739 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:36.084903 2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-191.ec2.internal\" not found" node="ip-10-0-131-191.ec2.internal" Jul 08 19:03:36.448739 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:36.166337 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-191.ec2.internal\" not found" Jul 08 19:03:36.448739 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:36.177874 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/330f763c6aa8f4d35b771e6847835d74-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-191.ec2.internal\" (UID: \"330f763c6aa8f4d35b771e6847835d74\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-191.ec2.internal" Jul 08 19:03:36.450123 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:36.177901 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ce1e796271b5d048919b6c245c7db2fc-config\") pod \"kube-apiserver-proxy-ip-10-0-131-191.ec2.internal\" (UID: \"ce1e796271b5d048919b6c245c7db2fc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-191.ec2.internal" Jul 08 19:03:36.450123 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:36.177928 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/330f763c6aa8f4d35b771e6847835d74-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-191.ec2.internal\" (UID: \"330f763c6aa8f4d35b771e6847835d74\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-191.ec2.internal" Jul 08 19:03:36.450123 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:36.177974 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ce1e796271b5d048919b6c245c7db2fc-config\") pod \"kube-apiserver-proxy-ip-10-0-131-191.ec2.internal\" (UID: \"ce1e796271b5d048919b6c245c7db2fc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-191.ec2.internal" Jul 08 19:03:36.450123 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:36.177979 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/330f763c6aa8f4d35b771e6847835d74-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-191.ec2.internal\" (UID: \"330f763c6aa8f4d35b771e6847835d74\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-191.ec2.internal" Jul 08 19:03:36.450123 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:36.178009 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/330f763c6aa8f4d35b771e6847835d74-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-191.ec2.internal\" (UID: \"330f763c6aa8f4d35b771e6847835d74\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-191.ec2.internal" Jul 08 19:03:36.450123 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:36.266621 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-191.ec2.internal\" not found" Jul 08 19:03:36.450123 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:36.366973 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-191.ec2.internal\" not found" Jul 08 19:03:36.450123 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:36.380893 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-191.ec2.internal" Jul 08 19:03:36.450123 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:36.388006 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-191.ec2.internal" Jul 08 19:03:36.633548 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:36.467717 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-191.ec2.internal\" not found" Jul 08 19:03:36.633548 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:36.568180 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-191.ec2.internal\" not found" Jul 08 19:03:38.184367 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:36.668643 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-191.ec2.internal\" not found" Jul 08 19:03:38.184367 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:38.064032 2558 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 08 19:03:38.184367 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:38.064159 2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 08 19:03:38.184367 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:38.120746 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-191.ec2.internal\" not found" Jul 08 19:03:38.184367 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:38.131336 2558 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 19:03:38.388153 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:38.194697 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 19:03:38.388153 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:38.195799 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-191.ec2.internal" event="NodeHasSufficientMemory" Jul 08 19:03:38.388153 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:38.195826 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-191.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 19:03:38.388153 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:38.195838 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-191.ec2.internal" event="NodeHasSufficientPID" Jul 08 19:03:38.388153 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:38.216139 2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-07 18:58:37 +0000 UTC" deadline="2028-04-05 04:02:24.993440912 +0000 UTC" Jul 08 19:03:38.388153 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:38.216167 2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15272h58m46.777277712s" Jul 08 19:03:38.388153 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:38.221247 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-191.ec2.internal\" not found" Jul 08 19:03:38.388153 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:38.316874 2558 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 19:03:38.388153 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:38.328154 2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-191.ec2.internal" Jul 08 19:03:38.388153 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:38.360096 2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 08 19:03:38.388153 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:38.362049 2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-191.ec2.internal" Jul 08 19:03:38.388153 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:38.382324 2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 08 19:03:38.623788 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:38.501024 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330f763c6aa8f4d35b771e6847835d74.slice/crio-8be439c23adb9a4cf36e6b5a07b6ee90d375bfb290c14d6f00712eef27247c17 WatchSource:0}: Error finding container 8be439c23adb9a4cf36e6b5a07b6ee90d375bfb290c14d6f00712eef27247c17: Status 404 returned error can't find the container with id 8be439c23adb9a4cf36e6b5a07b6ee90d375bfb290c14d6f00712eef27247c17 Jul 08 19:03:38.623788 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:38.501554 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce1e796271b5d048919b6c245c7db2fc.slice/crio-ac481461ecf98f5f5cf18ba0bc3085b6f03af650157af22642f81f5bc5019880 WatchSource:0}: Error finding container ac481461ecf98f5f5cf18ba0bc3085b6f03af650157af22642f81f5bc5019880: Status 404 returned error can't find the container with id ac481461ecf98f5f5cf18ba0bc3085b6f03af650157af22642f81f5bc5019880 Jul 08 19:03:38.623788 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:38.506862 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 19:03:38.623788 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:38.525785 2558 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 19:03:39.132756 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.101605 2558 apiserver.go:52] "Watching apiserver" Jul 08 19:03:39.132756 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.127890 2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 08 19:03:39.207266 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.152027 2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 19:03:39.207266 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.154480 2558 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 08 19:03:39.207266 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.154870 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glkc","openshift-cluster-node-tuning-operator/tuned-6mf6p","openshift-dns/node-resolver-44598","openshift-image-registry/node-ca-gktll","openshift-multus/multus-additional-cni-plugins-7ks2q","openshift-multus/network-metrics-daemon-5rmts","openshift-network-diagnostics/network-check-target-26v6h","kube-system/global-pull-secret-syncer-mlw2c","kube-system/konnectivity-agent-jwprt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-191.ec2.internal","openshift-multus/multus-rb7nx","openshift-network-operator/iptables-alerter-xcqj8","openshift-ovn-kubernetes/ovnkube-node-6gmjt","kube-system/kube-apiserver-proxy-ip-10-0-131-191.ec2.internal"] Jul 08 19:03:39.207266 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.157134 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26v6h" Jul 08 19:03:39.207266 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:39.157219 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26v6h" podUID="6fcd111d-1192-4c3e-90e7-2294cdec33ee" Jul 08 19:03:39.207266 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.159282 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.207266 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.161819 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-44598" Jul 08 19:03:39.207266 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.164102 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gktll" Jul 08 19:03:39.291972 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.240351 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7873f48e-334b-4a54-8908-191e668b953d-host-slash\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.291972 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.240377 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7873f48e-334b-4a54-8908-191e668b953d-var-lib-openvswitch\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.291972 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.240394 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7873f48e-334b-4a54-8908-191e668b953d-ovnkube-config\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.291972 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.240408 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7873f48e-334b-4a54-8908-191e668b953d-ovn-node-metrics-cert\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.291972 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.271155 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74f5d0c6-337e-407c-bdef-75c0dcd3c47d-tmp-dir\") pod \"node-resolver-44598\" (UID: \"74f5d0c6-337e-407c-bdef-75c0dcd3c47d\") " pod="openshift-dns/node-resolver-44598" Jul 08 19:03:39.291972 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.271210 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7873f48e-334b-4a54-8908-191e668b953d-host-kubelet\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.291972 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.273975 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7ks2q" Jul 08 19:03:39.291972 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.275067 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7873f48e-334b-4a54-8908-191e668b953d-systemd-units\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.291972 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.275094 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7873f48e-334b-4a54-8908-191e668b953d-host-run-ovn-kubernetes\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.291972 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.275121 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7873f48e-334b-4a54-8908-191e668b953d-ovnkube-script-lib\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.291972 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.275145 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a0658e5c-4e22-43d3-ac11-86df5ff39969-host\") pod \"node-ca-gktll\" (UID: \"a0658e5c-4e22-43d3-ac11-86df5ff39969\") " pod="openshift-image-registry/node-ca-gktll" Jul 08 19:03:39.291972 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.275170 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/60145409-2aa4-4d68-9405-30b21fae674e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7ks2q\" (UID: \"60145409-2aa4-4d68-9405-30b21fae674e\") " pod="openshift-multus/multus-additional-cni-plugins-7ks2q" Jul 08 19:03:39.291972 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.275196 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7873f48e-334b-4a54-8908-191e668b953d-log-socket\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.291972 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.275222 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7873f48e-334b-4a54-8908-191e668b953d-host-cni-netd\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.293618 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.275245 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60145409-2aa4-4d68-9405-30b21fae674e-system-cni-dir\") pod \"multus-additional-cni-plugins-7ks2q\" (UID: \"60145409-2aa4-4d68-9405-30b21fae674e\") " pod="openshift-multus/multus-additional-cni-plugins-7ks2q" Jul 08 19:03:39.293618 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.275269 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7873f48e-334b-4a54-8908-191e668b953d-run-systemd\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.293618 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.275379 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grmf9\" (UniqueName: \"kubernetes.io/projected/74f5d0c6-337e-407c-bdef-75c0dcd3c47d-kube-api-access-grmf9\") pod \"node-resolver-44598\" (UID: \"74f5d0c6-337e-407c-bdef-75c0dcd3c47d\") " pod="openshift-dns/node-resolver-44598" Jul 08 19:03:39.293618 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.275405 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9rp4\" (UniqueName: \"kubernetes.io/projected/a0658e5c-4e22-43d3-ac11-86df5ff39969-kube-api-access-r9rp4\") pod \"node-ca-gktll\" (UID: \"a0658e5c-4e22-43d3-ac11-86df5ff39969\") " pod="openshift-image-registry/node-ca-gktll" Jul 08 19:03:39.293618 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.275434 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/60145409-2aa4-4d68-9405-30b21fae674e-os-release\") pod \"multus-additional-cni-plugins-7ks2q\" (UID: \"60145409-2aa4-4d68-9405-30b21fae674e\") " pod="openshift-multus/multus-additional-cni-plugins-7ks2q" Jul 08 19:03:39.293618 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.275471 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7873f48e-334b-4a54-8908-191e668b953d-host-cni-bin\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.293618 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.275491 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/74f5d0c6-337e-407c-bdef-75c0dcd3c47d-hosts-file\") pod \"node-resolver-44598\" (UID: \"74f5d0c6-337e-407c-bdef-75c0dcd3c47d\") " pod="openshift-dns/node-resolver-44598" Jul 08 19:03:39.293618 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.275530 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60145409-2aa4-4d68-9405-30b21fae674e-cnibin\") pod \"multus-additional-cni-plugins-7ks2q\" (UID: \"60145409-2aa4-4d68-9405-30b21fae674e\") " pod="openshift-multus/multus-additional-cni-plugins-7ks2q" Jul 08 19:03:39.293618 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.275570 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60145409-2aa4-4d68-9405-30b21fae674e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7ks2q\" (UID: \"60145409-2aa4-4d68-9405-30b21fae674e\") " pod="openshift-multus/multus-additional-cni-plugins-7ks2q" Jul 08 19:03:39.293618 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.275614 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7873f48e-334b-4a54-8908-191e668b953d-host-run-netns\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.293618 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.275646 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7873f48e-334b-4a54-8908-191e668b953d-run-ovn\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.293618 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.275671 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7873f48e-334b-4a54-8908-191e668b953d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.293618 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.275689 2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 08 19:03:39.293618 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.275718 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7873f48e-334b-4a54-8908-191e668b953d-run-openvswitch\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.294479 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.275759 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7873f48e-334b-4a54-8908-191e668b953d-node-log\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.294479 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.275784 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7873f48e-334b-4a54-8908-191e668b953d-env-overrides\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.294479 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.275808 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvmw8\" (UniqueName: \"kubernetes.io/projected/7873f48e-334b-4a54-8908-191e668b953d-kube-api-access-vvmw8\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.294479 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.275834 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/60145409-2aa4-4d68-9405-30b21fae674e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7ks2q\" (UID: \"60145409-2aa4-4d68-9405-30b21fae674e\") " pod="openshift-multus/multus-additional-cni-plugins-7ks2q" Jul 08 19:03:39.294479 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.275859 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7873f48e-334b-4a54-8908-191e668b953d-etc-openvswitch\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.294479 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.275882 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a0658e5c-4e22-43d3-ac11-86df5ff39969-serviceca\") pod \"node-ca-gktll\" (UID: \"a0658e5c-4e22-43d3-ac11-86df5ff39969\") " pod="openshift-image-registry/node-ca-gktll" Jul 08 19:03:39.294479 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.275914 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60145409-2aa4-4d68-9405-30b21fae674e-cni-binary-copy\") pod \"multus-additional-cni-plugins-7ks2q\" (UID: \"60145409-2aa4-4d68-9405-30b21fae674e\") " pod="openshift-multus/multus-additional-cni-plugins-7ks2q" Jul 08 19:03:39.294479 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.275942 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ps6w\" (UniqueName: \"kubernetes.io/projected/60145409-2aa4-4d68-9405-30b21fae674e-kube-api-access-6ps6w\") pod \"multus-additional-cni-plugins-7ks2q\" (UID: \"60145409-2aa4-4d68-9405-30b21fae674e\") " pod="openshift-multus/multus-additional-cni-plugins-7ks2q" Jul 08 19:03:39.294479 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.275988 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lszs7\" (UniqueName: \"kubernetes.io/projected/6fcd111d-1192-4c3e-90e7-2294cdec33ee-kube-api-access-lszs7\") pod \"network-check-target-26v6h\" (UID: \"6fcd111d-1192-4c3e-90e7-2294cdec33ee\") " pod="openshift-network-diagnostics/network-check-target-26v6h" Jul 08 19:03:39.294479 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.276192 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5rmts" Jul 08 19:03:39.294479 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.276204 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-75qjl\"" Jul 08 19:03:39.294479 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:39.276259 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5rmts" podUID="99034b7e-72e2-4a28-8711-233630b7e9ea" Jul 08 19:03:39.294479 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.278482 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jwprt" Jul 08 19:03:39.294479 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.279902 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 08 19:03:39.294479 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.281002 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glkc" Jul 08 19:03:39.294479 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.283170 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" Jul 08 19:03:39.294479 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.284953 2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-dwdd5" Jul 08 19:03:39.294479 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.285045 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 08 19:03:39.296221 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.286826 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 08 19:03:39.296221 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.288898 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xcqj8" Jul 08 19:03:39.296221 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.292674 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.296221 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.295115 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-191.ec2.internal" event={"ID":"ce1e796271b5d048919b6c245c7db2fc","Type":"ContainerStarted","Data":"ac481461ecf98f5f5cf18ba0bc3085b6f03af650157af22642f81f5bc5019880"} Jul 08 19:03:39.296221 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.295303 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlw2c" Jul 08 19:03:39.296221 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:39.295367 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlw2c" podUID="640044e9-7c83-4f5a-9c9d-2e4183284806" Jul 08 19:03:39.302638 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.299004 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-191.ec2.internal" event={"ID":"330f763c6aa8f4d35b771e6847835d74","Type":"ContainerStarted","Data":"8be439c23adb9a4cf36e6b5a07b6ee90d375bfb290c14d6f00712eef27247c17"} Jul 08 19:03:39.331204 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.329299 2558 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 08 19:03:39.391172 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.377204 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7873f48e-334b-4a54-8908-191e668b953d-host-slash\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.391172 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.377244 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7873f48e-334b-4a54-8908-191e668b953d-var-lib-openvswitch\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.391172 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.377274 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1318ccd-51f3-4a63-9a79-28ebe1c65913-system-cni-dir\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.391172 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.377298 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1318ccd-51f3-4a63-9a79-28ebe1c65913-cni-binary-copy\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.391172 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.377322 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/17f1251c-a915-41c2-a44a-fd7a048826af-agent-certs\") pod \"konnectivity-agent-jwprt\" (UID: \"17f1251c-a915-41c2-a44a-fd7a048826af\") " pod="kube-system/konnectivity-agent-jwprt" Jul 08 19:03:39.391172 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.377345 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/17f1251c-a915-41c2-a44a-fd7a048826af-konnectivity-ca\") pod \"konnectivity-agent-jwprt\" (UID: \"17f1251c-a915-41c2-a44a-fd7a048826af\") " pod="kube-system/konnectivity-agent-jwprt" Jul 08 19:03:39.391172 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.377371 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7873f48e-334b-4a54-8908-191e668b953d-host-kubelet\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.391172 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.377394 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7873f48e-334b-4a54-8908-191e668b953d-host-run-ovn-kubernetes\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.391172 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.377436 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7873f48e-334b-4a54-8908-191e668b953d-ovnkube-script-lib\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.391172 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.377460 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99034b7e-72e2-4a28-8711-233630b7e9ea-metrics-certs\") pod \"network-metrics-daemon-5rmts\" (UID: \"99034b7e-72e2-4a28-8711-233630b7e9ea\") " pod="openshift-multus/network-metrics-daemon-5rmts" Jul 08 19:03:39.391172 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.377484 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1318ccd-51f3-4a63-9a79-28ebe1c65913-multus-conf-dir\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.391172 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.377507 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7873f48e-334b-4a54-8908-191e668b953d-log-socket\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.391172 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.377529 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a1318ccd-51f3-4a63-9a79-28ebe1c65913-hostroot\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.391172 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.377551 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/640044e9-7c83-4f5a-9c9d-2e4183284806-dbus\") pod \"global-pull-secret-syncer-mlw2c\" (UID: \"640044e9-7c83-4f5a-9c9d-2e4183284806\") " pod="kube-system/global-pull-secret-syncer-mlw2c" Jul 08 19:03:39.393594 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.377600 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9rp4\" (UniqueName: \"kubernetes.io/projected/a0658e5c-4e22-43d3-ac11-86df5ff39969-kube-api-access-r9rp4\") pod \"node-ca-gktll\" (UID: \"a0658e5c-4e22-43d3-ac11-86df5ff39969\") " pod="openshift-image-registry/node-ca-gktll" Jul 08 19:03:39.393594 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.377625 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/60145409-2aa4-4d68-9405-30b21fae674e-os-release\") pod \"multus-additional-cni-plugins-7ks2q\" (UID: \"60145409-2aa4-4d68-9405-30b21fae674e\") " pod="openshift-multus/multus-additional-cni-plugins-7ks2q" Jul 08 19:03:39.393594 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.377648 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc196626-817d-41e0-8bac-8c0ac61a559f-host\") pod \"tuned-6mf6p\" (UID: \"fc196626-817d-41e0-8bac-8c0ac61a559f\") " pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" Jul 08 19:03:39.393594 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.377678 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7873f48e-334b-4a54-8908-191e668b953d-host-cni-bin\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.393594 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.377702 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/74f5d0c6-337e-407c-bdef-75c0dcd3c47d-hosts-file\") pod \"node-resolver-44598\" (UID: \"74f5d0c6-337e-407c-bdef-75c0dcd3c47d\") " pod="openshift-dns/node-resolver-44598" Jul 08 19:03:39.393594 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.377726 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/903e7441-02b7-49ee-a946-75875b1e34e3-sys-fs\") pod \"aws-ebs-csi-driver-node-9glkc\" (UID: \"903e7441-02b7-49ee-a946-75875b1e34e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glkc" Jul 08 19:03:39.393594 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.377750 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n5bt\" (UniqueName: \"kubernetes.io/projected/903e7441-02b7-49ee-a946-75875b1e34e3-kube-api-access-4n5bt\") pod \"aws-ebs-csi-driver-node-9glkc\" (UID: \"903e7441-02b7-49ee-a946-75875b1e34e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glkc" Jul 08 19:03:39.393594 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.377776 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7873f48e-334b-4a54-8908-191e668b953d-host-run-netns\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.393594 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.377798 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7873f48e-334b-4a54-8908-191e668b953d-run-ovn\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.393594 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.377822 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a1318ccd-51f3-4a63-9a79-28ebe1c65913-multus-socket-dir-parent\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.393594 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.377845 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/903e7441-02b7-49ee-a946-75875b1e34e3-registration-dir\") pod \"aws-ebs-csi-driver-node-9glkc\" (UID: \"903e7441-02b7-49ee-a946-75875b1e34e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glkc" Jul 08 19:03:39.393594 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.377870 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7873f48e-334b-4a54-8908-191e668b953d-run-openvswitch\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.393594 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.377894 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7873f48e-334b-4a54-8908-191e668b953d-node-log\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.393594 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.377923 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7873f48e-334b-4a54-8908-191e668b953d-env-overrides\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.394863 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.377947 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvmw8\" (UniqueName: \"kubernetes.io/projected/7873f48e-334b-4a54-8908-191e668b953d-kube-api-access-vvmw8\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.394863 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.377975 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/60145409-2aa4-4d68-9405-30b21fae674e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7ks2q\" (UID: \"60145409-2aa4-4d68-9405-30b21fae674e\") " pod="openshift-multus/multus-additional-cni-plugins-7ks2q" Jul 08 19:03:39.394863 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.377997 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fc196626-817d-41e0-8bac-8c0ac61a559f-lib-modules\") pod \"tuned-6mf6p\" (UID: \"fc196626-817d-41e0-8bac-8c0ac61a559f\") " pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" Jul 08 19:03:39.394863 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.378021 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2px8t\" (UniqueName: \"kubernetes.io/projected/fc196626-817d-41e0-8bac-8c0ac61a559f-kube-api-access-2px8t\") pod \"tuned-6mf6p\" (UID: \"fc196626-817d-41e0-8bac-8c0ac61a559f\") " pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" Jul 08 19:03:39.394863 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.378049 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a1318ccd-51f3-4a63-9a79-28ebe1c65913-host-var-lib-cni-multus\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.394863 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.378074 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a0658e5c-4e22-43d3-ac11-86df5ff39969-serviceca\") pod \"node-ca-gktll\" (UID: \"a0658e5c-4e22-43d3-ac11-86df5ff39969\") " pod="openshift-image-registry/node-ca-gktll" Jul 08 19:03:39.394863 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.378113 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60145409-2aa4-4d68-9405-30b21fae674e-cni-binary-copy\") pod \"multus-additional-cni-plugins-7ks2q\" (UID: \"60145409-2aa4-4d68-9405-30b21fae674e\") " pod="openshift-multus/multus-additional-cni-plugins-7ks2q" Jul 08 19:03:39.394863 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.378138 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fc196626-817d-41e0-8bac-8c0ac61a559f-etc-sysctl-d\") pod \"tuned-6mf6p\" (UID: \"fc196626-817d-41e0-8bac-8c0ac61a559f\") " pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" Jul 08 19:03:39.394863 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.378166 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fc196626-817d-41e0-8bac-8c0ac61a559f-run\") pod \"tuned-6mf6p\" (UID: \"fc196626-817d-41e0-8bac-8c0ac61a559f\") " pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" Jul 08 19:03:39.394863 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.378190 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a1318ccd-51f3-4a63-9a79-28ebe1c65913-host-run-netns\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.394863 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.378216 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/640044e9-7c83-4f5a-9c9d-2e4183284806-kubelet-config\") pod \"global-pull-secret-syncer-mlw2c\" (UID: \"640044e9-7c83-4f5a-9c9d-2e4183284806\") " pod="kube-system/global-pull-secret-syncer-mlw2c" Jul 08 19:03:39.394863 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.378239 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/903e7441-02b7-49ee-a946-75875b1e34e3-device-dir\") pod \"aws-ebs-csi-driver-node-9glkc\" (UID: \"903e7441-02b7-49ee-a946-75875b1e34e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glkc" Jul 08 19:03:39.394863 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.378267 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/903e7441-02b7-49ee-a946-75875b1e34e3-etc-selinux\") pod \"aws-ebs-csi-driver-node-9glkc\" (UID: \"903e7441-02b7-49ee-a946-75875b1e34e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glkc" Jul 08 19:03:39.394863 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.378295 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7873f48e-334b-4a54-8908-191e668b953d-ovnkube-config\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.398887 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.378321 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7873f48e-334b-4a54-8908-191e668b953d-ovn-node-metrics-cert\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.398887 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.378346 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74f5d0c6-337e-407c-bdef-75c0dcd3c47d-tmp-dir\") pod \"node-resolver-44598\" (UID: \"74f5d0c6-337e-407c-bdef-75c0dcd3c47d\") " pod="openshift-dns/node-resolver-44598" Jul 08 19:03:39.398887 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.378370 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd5c6\" (UniqueName: \"kubernetes.io/projected/99034b7e-72e2-4a28-8711-233630b7e9ea-kube-api-access-zd5c6\") pod \"network-metrics-daemon-5rmts\" (UID: \"99034b7e-72e2-4a28-8711-233630b7e9ea\") " pod="openshift-multus/network-metrics-daemon-5rmts" Jul 08 19:03:39.398887 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.378393 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a1318ccd-51f3-4a63-9a79-28ebe1c65913-os-release\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.398887 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.378416 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a1318ccd-51f3-4a63-9a79-28ebe1c65913-host-var-lib-cni-bin\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.398887 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.378440 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1318ccd-51f3-4a63-9a79-28ebe1c65913-host-var-lib-kubelet\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.398887 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.378464 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1318ccd-51f3-4a63-9a79-28ebe1c65913-etc-kubernetes\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.398887 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.378493 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7873f48e-334b-4a54-8908-191e668b953d-systemd-units\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.398887 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.378519 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a0658e5c-4e22-43d3-ac11-86df5ff39969-host\") pod \"node-ca-gktll\" (UID: \"a0658e5c-4e22-43d3-ac11-86df5ff39969\") " pod="openshift-image-registry/node-ca-gktll" Jul 08 19:03:39.398887 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.378551 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/60145409-2aa4-4d68-9405-30b21fae674e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7ks2q\" (UID: \"60145409-2aa4-4d68-9405-30b21fae674e\") " pod="openshift-multus/multus-additional-cni-plugins-7ks2q" Jul 08 19:03:39.398887 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.378591 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fc196626-817d-41e0-8bac-8c0ac61a559f-etc-sysctl-conf\") pod \"tuned-6mf6p\" (UID: \"fc196626-817d-41e0-8bac-8c0ac61a559f\") " pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" Jul 08 19:03:39.398887 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.378615 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fc196626-817d-41e0-8bac-8c0ac61a559f-etc-systemd\") pod \"tuned-6mf6p\" (UID: \"fc196626-817d-41e0-8bac-8c0ac61a559f\") " pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" Jul 08 19:03:39.398887 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.378638 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1318ccd-51f3-4a63-9a79-28ebe1c65913-cnibin\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.398887 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.378679 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a1318ccd-51f3-4a63-9a79-28ebe1c65913-multus-daemon-config\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.399997 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.378704 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/640044e9-7c83-4f5a-9c9d-2e4183284806-original-pull-secret\") pod \"global-pull-secret-syncer-mlw2c\" (UID: \"640044e9-7c83-4f5a-9c9d-2e4183284806\") " pod="kube-system/global-pull-secret-syncer-mlw2c" Jul 08 19:03:39.399997 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.378731 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7873f48e-334b-4a54-8908-191e668b953d-host-cni-netd\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.399997 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.378756 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60145409-2aa4-4d68-9405-30b21fae674e-system-cni-dir\") pod \"multus-additional-cni-plugins-7ks2q\" (UID: \"60145409-2aa4-4d68-9405-30b21fae674e\") " pod="openshift-multus/multus-additional-cni-plugins-7ks2q" Jul 08 19:03:39.399997 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.378779 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fc196626-817d-41e0-8bac-8c0ac61a559f-etc-tuned\") pod \"tuned-6mf6p\" (UID: \"fc196626-817d-41e0-8bac-8c0ac61a559f\") " pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" Jul 08 19:03:39.399997 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.378802 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pd2g\" (UniqueName: \"kubernetes.io/projected/a1318ccd-51f3-4a63-9a79-28ebe1c65913-kube-api-access-8pd2g\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.399997 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.378827 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/903e7441-02b7-49ee-a946-75875b1e34e3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9glkc\" (UID: \"903e7441-02b7-49ee-a946-75875b1e34e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glkc" Jul 08 19:03:39.399997 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.378863 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7873f48e-334b-4a54-8908-191e668b953d-run-systemd\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.399997 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.378890 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grmf9\" (UniqueName: \"kubernetes.io/projected/74f5d0c6-337e-407c-bdef-75c0dcd3c47d-kube-api-access-grmf9\") pod \"node-resolver-44598\" (UID: \"74f5d0c6-337e-407c-bdef-75c0dcd3c47d\") " pod="openshift-dns/node-resolver-44598" Jul 08 19:03:39.399997 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.378914 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc196626-817d-41e0-8bac-8c0ac61a559f-var-lib-kubelet\") pod \"tuned-6mf6p\" (UID: \"fc196626-817d-41e0-8bac-8c0ac61a559f\") " pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" Jul 08 19:03:39.399997 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.378940 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/903e7441-02b7-49ee-a946-75875b1e34e3-socket-dir\") pod \"aws-ebs-csi-driver-node-9glkc\" (UID: \"903e7441-02b7-49ee-a946-75875b1e34e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glkc" Jul 08 19:03:39.399997 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.378966 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60145409-2aa4-4d68-9405-30b21fae674e-cnibin\") pod \"multus-additional-cni-plugins-7ks2q\" (UID: \"60145409-2aa4-4d68-9405-30b21fae674e\") " pod="openshift-multus/multus-additional-cni-plugins-7ks2q" Jul 08 19:03:39.399997 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.378991 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60145409-2aa4-4d68-9405-30b21fae674e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7ks2q\" (UID: \"60145409-2aa4-4d68-9405-30b21fae674e\") " pod="openshift-multus/multus-additional-cni-plugins-7ks2q" Jul 08 19:03:39.399997 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.379021 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fc196626-817d-41e0-8bac-8c0ac61a559f-etc-modprobe-d\") pod \"tuned-6mf6p\" (UID: \"fc196626-817d-41e0-8bac-8c0ac61a559f\") " pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" Jul 08 19:03:39.399997 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.379044 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc196626-817d-41e0-8bac-8c0ac61a559f-tmp\") pod \"tuned-6mf6p\" (UID: \"fc196626-817d-41e0-8bac-8c0ac61a559f\") " pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" Jul 08 19:03:39.404176 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.379067 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mctt2\" (UniqueName: \"kubernetes.io/projected/9958d823-d456-441f-9bef-868175bf2db8-kube-api-access-mctt2\") pod \"iptables-alerter-xcqj8\" (UID: \"9958d823-d456-441f-9bef-868175bf2db8\") " pod="openshift-network-operator/iptables-alerter-xcqj8" Jul 08 19:03:39.404176 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.379091 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1318ccd-51f3-4a63-9a79-28ebe1c65913-multus-cni-dir\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.404176 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.379115 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a1318ccd-51f3-4a63-9a79-28ebe1c65913-host-run-k8s-cni-cncf-io\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.404176 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.379143 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7873f48e-334b-4a54-8908-191e668b953d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.404176 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.379170 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc196626-817d-41e0-8bac-8c0ac61a559f-etc-kubernetes\") pod \"tuned-6mf6p\" (UID: \"fc196626-817d-41e0-8bac-8c0ac61a559f\") " pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" Jul 08 19:03:39.404176 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.379192 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc196626-817d-41e0-8bac-8c0ac61a559f-sys\") pod \"tuned-6mf6p\" (UID: \"fc196626-817d-41e0-8bac-8c0ac61a559f\") " pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" Jul 08 19:03:39.404176 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.379218 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a1318ccd-51f3-4a63-9a79-28ebe1c65913-host-run-multus-certs\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.404176 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.379243 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fc196626-817d-41e0-8bac-8c0ac61a559f-etc-sysconfig\") pod \"tuned-6mf6p\" (UID: \"fc196626-817d-41e0-8bac-8c0ac61a559f\") " pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" Jul 08 19:03:39.404176 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.379301 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7873f48e-334b-4a54-8908-191e668b953d-etc-openvswitch\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.404176 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.379329 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ps6w\" (UniqueName: \"kubernetes.io/projected/60145409-2aa4-4d68-9405-30b21fae674e-kube-api-access-6ps6w\") pod \"multus-additional-cni-plugins-7ks2q\" (UID: \"60145409-2aa4-4d68-9405-30b21fae674e\") " pod="openshift-multus/multus-additional-cni-plugins-7ks2q" Jul 08 19:03:39.404176 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.379358 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lszs7\" (UniqueName: \"kubernetes.io/projected/6fcd111d-1192-4c3e-90e7-2294cdec33ee-kube-api-access-lszs7\") pod \"network-check-target-26v6h\" (UID: \"6fcd111d-1192-4c3e-90e7-2294cdec33ee\") " pod="openshift-network-diagnostics/network-check-target-26v6h" Jul 08 19:03:39.404176 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.379388 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9958d823-d456-441f-9bef-868175bf2db8-iptables-alerter-script\") pod \"iptables-alerter-xcqj8\" (UID: \"9958d823-d456-441f-9bef-868175bf2db8\") " pod="openshift-network-operator/iptables-alerter-xcqj8" Jul 08 19:03:39.404176 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.379417 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9958d823-d456-441f-9bef-868175bf2db8-host-slash\") pod \"iptables-alerter-xcqj8\" (UID: \"9958d823-d456-441f-9bef-868175bf2db8\") " pod="openshift-network-operator/iptables-alerter-xcqj8" Jul 08 19:03:39.404176 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.379513 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7873f48e-334b-4a54-8908-191e668b953d-host-slash\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.405023 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.379613 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7873f48e-334b-4a54-8908-191e668b953d-var-lib-openvswitch\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.405023 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.379697 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7873f48e-334b-4a54-8908-191e668b953d-host-kubelet\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.405023 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.379740 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7873f48e-334b-4a54-8908-191e668b953d-host-run-ovn-kubernetes\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.405023 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.380340 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7873f48e-334b-4a54-8908-191e668b953d-ovnkube-script-lib\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.405023 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.380446 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7873f48e-334b-4a54-8908-191e668b953d-log-socket\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.405023 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.380843 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/60145409-2aa4-4d68-9405-30b21fae674e-os-release\") pod \"multus-additional-cni-plugins-7ks2q\" (UID: \"60145409-2aa4-4d68-9405-30b21fae674e\") " pod="openshift-multus/multus-additional-cni-plugins-7ks2q" Jul 08 19:03:39.405023 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.380904 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7873f48e-334b-4a54-8908-191e668b953d-host-cni-bin\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.405023 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.380955 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/74f5d0c6-337e-407c-bdef-75c0dcd3c47d-hosts-file\") pod \"node-resolver-44598\" (UID: \"74f5d0c6-337e-407c-bdef-75c0dcd3c47d\") " pod="openshift-dns/node-resolver-44598" Jul 08 19:03:39.405023 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.381015 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7873f48e-334b-4a54-8908-191e668b953d-host-run-netns\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.405023 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.381059 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7873f48e-334b-4a54-8908-191e668b953d-run-ovn\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.405023 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.381115 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7873f48e-334b-4a54-8908-191e668b953d-run-openvswitch\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.405023 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.381156 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7873f48e-334b-4a54-8908-191e668b953d-node-log\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.405023 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.381520 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7873f48e-334b-4a54-8908-191e668b953d-env-overrides\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.405023 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.382287 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74f5d0c6-337e-407c-bdef-75c0dcd3c47d-tmp-dir\") pod \"node-resolver-44598\" (UID: \"74f5d0c6-337e-407c-bdef-75c0dcd3c47d\") " pod="openshift-dns/node-resolver-44598" Jul 08 19:03:39.405023 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.382416 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7873f48e-334b-4a54-8908-191e668b953d-systemd-units\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.406127 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.382463 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a0658e5c-4e22-43d3-ac11-86df5ff39969-host\") pod \"node-ca-gktll\" (UID: \"a0658e5c-4e22-43d3-ac11-86df5ff39969\") " pod="openshift-image-registry/node-ca-gktll" Jul 08 19:03:39.406127 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.382607 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7873f48e-334b-4a54-8908-191e668b953d-host-cni-netd\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.406127 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.382650 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60145409-2aa4-4d68-9405-30b21fae674e-system-cni-dir\") pod \"multus-additional-cni-plugins-7ks2q\" (UID: \"60145409-2aa4-4d68-9405-30b21fae674e\") " pod="openshift-multus/multus-additional-cni-plugins-7ks2q" Jul 08 19:03:39.406127 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.382720 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7873f48e-334b-4a54-8908-191e668b953d-run-systemd\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.406127 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.382958 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60145409-2aa4-4d68-9405-30b21fae674e-cnibin\") pod \"multus-additional-cni-plugins-7ks2q\" (UID: \"60145409-2aa4-4d68-9405-30b21fae674e\") " pod="openshift-multus/multus-additional-cni-plugins-7ks2q" Jul 08 19:03:39.406127 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.383097 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60145409-2aa4-4d68-9405-30b21fae674e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7ks2q\" (UID: \"60145409-2aa4-4d68-9405-30b21fae674e\") " pod="openshift-multus/multus-additional-cni-plugins-7ks2q" Jul 08 19:03:39.406127 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.383183 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7873f48e-334b-4a54-8908-191e668b953d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.406127 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.383232 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7873f48e-334b-4a54-8908-191e668b953d-etc-openvswitch\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.534053 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.477675 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 08 19:03:39.534053 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.477732 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 08 19:03:39.534053 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.477780 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 08 19:03:39.534053 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.477890 2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-dwdd5" Jul 08 19:03:39.534053 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.477952 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 08 19:03:39.534053 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.477974 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 08 19:03:39.534053 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.478028 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 08 19:03:39.534053 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.478032 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jpmpp\"" Jul 08 19:03:39.534053 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.477978 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 08 19:03:39.534053 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.478691 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 08 19:03:39.534053 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.478704 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 08 19:03:39.534053 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.478742 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 08 19:03:39.534053 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.478750 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 08 19:03:39.534053 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.478785 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 08 19:03:39.534053 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.479103 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 08 19:03:39.534053 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.479183 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 08 19:03:39.534053 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.479250 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-crn2m\"" Jul 08 19:03:39.534053 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.479108 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 08 19:03:39.534053 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.479384 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 08 19:03:39.534053 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.479416 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 08 19:03:39.534053 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.479529 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 08 19:03:39.534053 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.479637 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kxdnd\"" Jul 08 19:03:39.534053 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.479737 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 08 19:03:39.534053 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.479878 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-z6zlx\"" Jul 08 19:03:39.534053 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.479984 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 08 19:03:39.534053 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.479994 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pvkzb\"" Jul 08 19:03:39.534053 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.480058 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mvht8\"" Jul 08 19:03:39.534053 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.480497 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fc196626-817d-41e0-8bac-8c0ac61a559f-lib-modules\") pod \"tuned-6mf6p\" (UID: \"fc196626-817d-41e0-8bac-8c0ac61a559f\") " pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" Jul 08 19:03:39.534053 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.480525 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fc196626-817d-41e0-8bac-8c0ac61a559f-lib-modules\") pod \"tuned-6mf6p\" (UID: \"fc196626-817d-41e0-8bac-8c0ac61a559f\") " pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" Jul 08 19:03:39.535529 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.481388 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2px8t\" (UniqueName: \"kubernetes.io/projected/fc196626-817d-41e0-8bac-8c0ac61a559f-kube-api-access-2px8t\") pod \"tuned-6mf6p\" (UID: \"fc196626-817d-41e0-8bac-8c0ac61a559f\") " pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" Jul 08 19:03:39.535529 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.481835 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a1318ccd-51f3-4a63-9a79-28ebe1c65913-host-var-lib-cni-multus\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.535529 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.481862 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fc196626-817d-41e0-8bac-8c0ac61a559f-etc-sysctl-d\") pod \"tuned-6mf6p\" (UID: \"fc196626-817d-41e0-8bac-8c0ac61a559f\") " pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" Jul 08 19:03:39.535529 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.481886 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fc196626-817d-41e0-8bac-8c0ac61a559f-run\") pod \"tuned-6mf6p\" (UID: \"fc196626-817d-41e0-8bac-8c0ac61a559f\") " pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" Jul 08 19:03:39.535529 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.481909 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a1318ccd-51f3-4a63-9a79-28ebe1c65913-host-run-netns\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.535529 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.481942 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/640044e9-7c83-4f5a-9c9d-2e4183284806-kubelet-config\") pod \"global-pull-secret-syncer-mlw2c\" (UID: \"640044e9-7c83-4f5a-9c9d-2e4183284806\") " pod="kube-system/global-pull-secret-syncer-mlw2c" Jul 08 19:03:39.535529 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.481966 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/903e7441-02b7-49ee-a946-75875b1e34e3-device-dir\") pod \"aws-ebs-csi-driver-node-9glkc\" (UID: \"903e7441-02b7-49ee-a946-75875b1e34e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glkc" Jul 08 19:03:39.535529 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.481987 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/903e7441-02b7-49ee-a946-75875b1e34e3-etc-selinux\") pod \"aws-ebs-csi-driver-node-9glkc\" (UID: \"903e7441-02b7-49ee-a946-75875b1e34e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glkc" Jul 08 19:03:39.535529 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482012 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd5c6\" (UniqueName: \"kubernetes.io/projected/99034b7e-72e2-4a28-8711-233630b7e9ea-kube-api-access-zd5c6\") pod \"network-metrics-daemon-5rmts\" (UID: \"99034b7e-72e2-4a28-8711-233630b7e9ea\") " pod="openshift-multus/network-metrics-daemon-5rmts" Jul 08 19:03:39.535529 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482034 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a1318ccd-51f3-4a63-9a79-28ebe1c65913-os-release\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.535529 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482044 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fc196626-817d-41e0-8bac-8c0ac61a559f-etc-sysctl-d\") pod \"tuned-6mf6p\" (UID: \"fc196626-817d-41e0-8bac-8c0ac61a559f\") " pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" Jul 08 19:03:39.535529 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482057 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a1318ccd-51f3-4a63-9a79-28ebe1c65913-host-var-lib-cni-bin\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.535529 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482098 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1318ccd-51f3-4a63-9a79-28ebe1c65913-host-var-lib-kubelet\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.535529 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482123 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1318ccd-51f3-4a63-9a79-28ebe1c65913-etc-kubernetes\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.535529 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482119 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a1318ccd-51f3-4a63-9a79-28ebe1c65913-host-var-lib-cni-multus\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.536809 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482148 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fc196626-817d-41e0-8bac-8c0ac61a559f-etc-sysctl-conf\") pod \"tuned-6mf6p\" (UID: \"fc196626-817d-41e0-8bac-8c0ac61a559f\") " pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" Jul 08 19:03:39.536809 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482170 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fc196626-817d-41e0-8bac-8c0ac61a559f-etc-systemd\") pod \"tuned-6mf6p\" (UID: \"fc196626-817d-41e0-8bac-8c0ac61a559f\") " pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" Jul 08 19:03:39.536809 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482177 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/903e7441-02b7-49ee-a946-75875b1e34e3-etc-selinux\") pod \"aws-ebs-csi-driver-node-9glkc\" (UID: \"903e7441-02b7-49ee-a946-75875b1e34e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glkc" Jul 08 19:03:39.536809 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482193 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1318ccd-51f3-4a63-9a79-28ebe1c65913-cnibin\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.536809 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482216 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a1318ccd-51f3-4a63-9a79-28ebe1c65913-multus-daemon-config\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.536809 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482222 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a1318ccd-51f3-4a63-9a79-28ebe1c65913-host-run-netns\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.536809 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482240 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/640044e9-7c83-4f5a-9c9d-2e4183284806-original-pull-secret\") pod \"global-pull-secret-syncer-mlw2c\" (UID: \"640044e9-7c83-4f5a-9c9d-2e4183284806\") " pod="kube-system/global-pull-secret-syncer-mlw2c" Jul 08 19:03:39.536809 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482268 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fc196626-817d-41e0-8bac-8c0ac61a559f-etc-tuned\") pod \"tuned-6mf6p\" (UID: \"fc196626-817d-41e0-8bac-8c0ac61a559f\") " pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" Jul 08 19:03:39.536809 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482271 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/640044e9-7c83-4f5a-9c9d-2e4183284806-kubelet-config\") pod \"global-pull-secret-syncer-mlw2c\" (UID: \"640044e9-7c83-4f5a-9c9d-2e4183284806\") " pod="kube-system/global-pull-secret-syncer-mlw2c" Jul 08 19:03:39.536809 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482291 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pd2g\" (UniqueName: \"kubernetes.io/projected/a1318ccd-51f3-4a63-9a79-28ebe1c65913-kube-api-access-8pd2g\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.536809 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482316 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/903e7441-02b7-49ee-a946-75875b1e34e3-device-dir\") pod \"aws-ebs-csi-driver-node-9glkc\" (UID: \"903e7441-02b7-49ee-a946-75875b1e34e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glkc" Jul 08 19:03:39.536809 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482319 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/903e7441-02b7-49ee-a946-75875b1e34e3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9glkc\" (UID: \"903e7441-02b7-49ee-a946-75875b1e34e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glkc" Jul 08 19:03:39.536809 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482350 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/60145409-2aa4-4d68-9405-30b21fae674e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7ks2q\" (UID: \"60145409-2aa4-4d68-9405-30b21fae674e\") " pod="openshift-multus/multus-additional-cni-plugins-7ks2q" Jul 08 19:03:39.536809 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482357 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc196626-817d-41e0-8bac-8c0ac61a559f-var-lib-kubelet\") pod \"tuned-6mf6p\" (UID: \"fc196626-817d-41e0-8bac-8c0ac61a559f\") " pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" Jul 08 19:03:39.537494 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482380 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/903e7441-02b7-49ee-a946-75875b1e34e3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9glkc\" (UID: \"903e7441-02b7-49ee-a946-75875b1e34e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glkc" Jul 08 19:03:39.537494 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482391 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc196626-817d-41e0-8bac-8c0ac61a559f-var-lib-kubelet\") pod \"tuned-6mf6p\" (UID: \"fc196626-817d-41e0-8bac-8c0ac61a559f\") " pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" Jul 08 19:03:39.537494 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482411 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/903e7441-02b7-49ee-a946-75875b1e34e3-socket-dir\") pod \"aws-ebs-csi-driver-node-9glkc\" (UID: \"903e7441-02b7-49ee-a946-75875b1e34e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glkc" Jul 08 19:03:39.537494 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482429 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1318ccd-51f3-4a63-9a79-28ebe1c65913-host-var-lib-kubelet\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.537494 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482436 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1318ccd-51f3-4a63-9a79-28ebe1c65913-cnibin\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.537494 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482439 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fc196626-817d-41e0-8bac-8c0ac61a559f-etc-modprobe-d\") pod \"tuned-6mf6p\" (UID: \"fc196626-817d-41e0-8bac-8c0ac61a559f\") " pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" Jul 08 19:03:39.537494 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482465 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc196626-817d-41e0-8bac-8c0ac61a559f-tmp\") pod \"tuned-6mf6p\" (UID: \"fc196626-817d-41e0-8bac-8c0ac61a559f\") " pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" Jul 08 19:03:39.537494 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482405 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a1318ccd-51f3-4a63-9a79-28ebe1c65913-os-release\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.537494 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482477 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1318ccd-51f3-4a63-9a79-28ebe1c65913-etc-kubernetes\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.537494 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482490 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mctt2\" (UniqueName: \"kubernetes.io/projected/9958d823-d456-441f-9bef-868175bf2db8-kube-api-access-mctt2\") pod \"iptables-alerter-xcqj8\" (UID: \"9958d823-d456-441f-9bef-868175bf2db8\") " pod="openshift-network-operator/iptables-alerter-xcqj8" Jul 08 19:03:39.537494 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482544 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1318ccd-51f3-4a63-9a79-28ebe1c65913-multus-cni-dir\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.537494 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482552 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fc196626-817d-41e0-8bac-8c0ac61a559f-etc-systemd\") pod \"tuned-6mf6p\" (UID: \"fc196626-817d-41e0-8bac-8c0ac61a559f\") " pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" Jul 08 19:03:39.537494 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482571 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a1318ccd-51f3-4a63-9a79-28ebe1c65913-host-run-k8s-cni-cncf-io\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.537494 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482598 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fc196626-817d-41e0-8bac-8c0ac61a559f-etc-sysctl-conf\") pod \"tuned-6mf6p\" (UID: \"fc196626-817d-41e0-8bac-8c0ac61a559f\") " pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" Jul 08 19:03:39.537494 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482601 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fc196626-817d-41e0-8bac-8c0ac61a559f-etc-modprobe-d\") pod \"tuned-6mf6p\" (UID: \"fc196626-817d-41e0-8bac-8c0ac61a559f\") " pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" Jul 08 19:03:39.539175 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482641 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a1318ccd-51f3-4a63-9a79-28ebe1c65913-host-var-lib-cni-bin\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.539175 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:39.482645 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 19:03:39.539175 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:39.482707 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/640044e9-7c83-4f5a-9c9d-2e4183284806-original-pull-secret podName:640044e9-7c83-4f5a-9c9d-2e4183284806 nodeName:}" failed. No retries permitted until 2026-07-08 19:03:39.982686384 +0000 UTC m=+3.241639307 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/640044e9-7c83-4f5a-9c9d-2e4183284806-original-pull-secret") pod "global-pull-secret-syncer-mlw2c" (UID: "640044e9-7c83-4f5a-9c9d-2e4183284806") : object "kube-system"/"original-pull-secret" not registered Jul 08 19:03:39.539175 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482735 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/903e7441-02b7-49ee-a946-75875b1e34e3-socket-dir\") pod \"aws-ebs-csi-driver-node-9glkc\" (UID: \"903e7441-02b7-49ee-a946-75875b1e34e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glkc" Jul 08 19:03:39.539175 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482765 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1318ccd-51f3-4a63-9a79-28ebe1c65913-multus-cni-dir\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.539175 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482797 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a1318ccd-51f3-4a63-9a79-28ebe1c65913-host-run-k8s-cni-cncf-io\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.539175 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482801 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc196626-817d-41e0-8bac-8c0ac61a559f-etc-kubernetes\") pod \"tuned-6mf6p\" (UID: \"fc196626-817d-41e0-8bac-8c0ac61a559f\") " pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" Jul 08 19:03:39.539175 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482834 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc196626-817d-41e0-8bac-8c0ac61a559f-sys\") pod \"tuned-6mf6p\" (UID: \"fc196626-817d-41e0-8bac-8c0ac61a559f\") " pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" Jul 08 19:03:39.539175 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482859 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a1318ccd-51f3-4a63-9a79-28ebe1c65913-host-run-multus-certs\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.539175 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482862 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc196626-817d-41e0-8bac-8c0ac61a559f-etc-kubernetes\") pod \"tuned-6mf6p\" (UID: \"fc196626-817d-41e0-8bac-8c0ac61a559f\") " pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" Jul 08 19:03:39.539175 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482800 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fc196626-817d-41e0-8bac-8c0ac61a559f-run\") pod \"tuned-6mf6p\" (UID: \"fc196626-817d-41e0-8bac-8c0ac61a559f\") " pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" Jul 08 19:03:39.539175 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482883 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fc196626-817d-41e0-8bac-8c0ac61a559f-etc-sysconfig\") pod \"tuned-6mf6p\" (UID: \"fc196626-817d-41e0-8bac-8c0ac61a559f\") " pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" Jul 08 19:03:39.539175 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482909 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9958d823-d456-441f-9bef-868175bf2db8-iptables-alerter-script\") pod \"iptables-alerter-xcqj8\" (UID: \"9958d823-d456-441f-9bef-868175bf2db8\") " pod="openshift-network-operator/iptables-alerter-xcqj8" Jul 08 19:03:39.539175 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482927 2558 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 08 19:03:39.539175 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482968 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9958d823-d456-441f-9bef-868175bf2db8-host-slash\") pod \"iptables-alerter-xcqj8\" (UID: \"9958d823-d456-441f-9bef-868175bf2db8\") " pod="openshift-network-operator/iptables-alerter-xcqj8" Jul 08 19:03:39.540271 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.483013 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc196626-817d-41e0-8bac-8c0ac61a559f-sys\") pod \"tuned-6mf6p\" (UID: \"fc196626-817d-41e0-8bac-8c0ac61a559f\") " pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" Jul 08 19:03:39.540271 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.483052 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a1318ccd-51f3-4a63-9a79-28ebe1c65913-host-run-multus-certs\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.540271 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.483097 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fc196626-817d-41e0-8bac-8c0ac61a559f-etc-sysconfig\") pod \"tuned-6mf6p\" (UID: \"fc196626-817d-41e0-8bac-8c0ac61a559f\") " pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" Jul 08 19:03:39.540271 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.483237 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a0658e5c-4e22-43d3-ac11-86df5ff39969-serviceca\") pod \"node-ca-gktll\" (UID: \"a0658e5c-4e22-43d3-ac11-86df5ff39969\") " pod="openshift-image-registry/node-ca-gktll" Jul 08 19:03:39.540271 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.483257 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/60145409-2aa4-4d68-9405-30b21fae674e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7ks2q\" (UID: \"60145409-2aa4-4d68-9405-30b21fae674e\") " pod="openshift-multus/multus-additional-cni-plugins-7ks2q" Jul 08 19:03:39.540271 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.482934 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9958d823-d456-441f-9bef-868175bf2db8-host-slash\") pod \"iptables-alerter-xcqj8\" (UID: \"9958d823-d456-441f-9bef-868175bf2db8\") " pod="openshift-network-operator/iptables-alerter-xcqj8" Jul 08 19:03:39.540271 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.483315 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1318ccd-51f3-4a63-9a79-28ebe1c65913-system-cni-dir\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.540271 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.483370 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60145409-2aa4-4d68-9405-30b21fae674e-cni-binary-copy\") pod \"multus-additional-cni-plugins-7ks2q\" (UID: \"60145409-2aa4-4d68-9405-30b21fae674e\") " pod="openshift-multus/multus-additional-cni-plugins-7ks2q" Jul 08 19:03:39.540271 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.483348 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1318ccd-51f3-4a63-9a79-28ebe1c65913-cni-binary-copy\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.540271 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.483405 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/17f1251c-a915-41c2-a44a-fd7a048826af-agent-certs\") pod \"konnectivity-agent-jwprt\" (UID: \"17f1251c-a915-41c2-a44a-fd7a048826af\") " pod="kube-system/konnectivity-agent-jwprt" Jul 08 19:03:39.540271 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.483406 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1318ccd-51f3-4a63-9a79-28ebe1c65913-system-cni-dir\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.540271 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.483424 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/17f1251c-a915-41c2-a44a-fd7a048826af-konnectivity-ca\") pod \"konnectivity-agent-jwprt\" (UID: \"17f1251c-a915-41c2-a44a-fd7a048826af\") " pod="kube-system/konnectivity-agent-jwprt" Jul 08 19:03:39.540271 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.483447 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99034b7e-72e2-4a28-8711-233630b7e9ea-metrics-certs\") pod \"network-metrics-daemon-5rmts\" (UID: \"99034b7e-72e2-4a28-8711-233630b7e9ea\") " pod="openshift-multus/network-metrics-daemon-5rmts" Jul 08 19:03:39.540271 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.483466 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1318ccd-51f3-4a63-9a79-28ebe1c65913-multus-conf-dir\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.540271 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.483487 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a1318ccd-51f3-4a63-9a79-28ebe1c65913-hostroot\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.541324 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.483505 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/640044e9-7c83-4f5a-9c9d-2e4183284806-dbus\") pod \"global-pull-secret-syncer-mlw2c\" (UID: \"640044e9-7c83-4f5a-9c9d-2e4183284806\") " pod="kube-system/global-pull-secret-syncer-mlw2c" Jul 08 19:03:39.541324 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.483531 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc196626-817d-41e0-8bac-8c0ac61a559f-host\") pod \"tuned-6mf6p\" (UID: \"fc196626-817d-41e0-8bac-8c0ac61a559f\") " pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" Jul 08 19:03:39.541324 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.483559 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/903e7441-02b7-49ee-a946-75875b1e34e3-sys-fs\") pod \"aws-ebs-csi-driver-node-9glkc\" (UID: \"903e7441-02b7-49ee-a946-75875b1e34e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glkc" Jul 08 19:03:39.541324 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.483614 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n5bt\" (UniqueName: \"kubernetes.io/projected/903e7441-02b7-49ee-a946-75875b1e34e3-kube-api-access-4n5bt\") pod \"aws-ebs-csi-driver-node-9glkc\" (UID: \"903e7441-02b7-49ee-a946-75875b1e34e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glkc" Jul 08 19:03:39.541324 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.483630 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7873f48e-334b-4a54-8908-191e668b953d-ovnkube-config\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.541324 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.483636 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a1318ccd-51f3-4a63-9a79-28ebe1c65913-multus-socket-dir-parent\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.541324 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.483662 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/903e7441-02b7-49ee-a946-75875b1e34e3-registration-dir\") pod \"aws-ebs-csi-driver-node-9glkc\" (UID: \"903e7441-02b7-49ee-a946-75875b1e34e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glkc" Jul 08 19:03:39.541324 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.483705 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/640044e9-7c83-4f5a-9c9d-2e4183284806-dbus\") pod \"global-pull-secret-syncer-mlw2c\" (UID: \"640044e9-7c83-4f5a-9c9d-2e4183284806\") " pod="kube-system/global-pull-secret-syncer-mlw2c" Jul 08 19:03:39.541324 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.483735 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc196626-817d-41e0-8bac-8c0ac61a559f-host\") pod \"tuned-6mf6p\" (UID: \"fc196626-817d-41e0-8bac-8c0ac61a559f\") " pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" Jul 08 19:03:39.541324 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.483779 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/903e7441-02b7-49ee-a946-75875b1e34e3-sys-fs\") pod \"aws-ebs-csi-driver-node-9glkc\" (UID: \"903e7441-02b7-49ee-a946-75875b1e34e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glkc" Jul 08 19:03:39.541324 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.483862 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a1318ccd-51f3-4a63-9a79-28ebe1c65913-multus-socket-dir-parent\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.541324 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.483913 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1318ccd-51f3-4a63-9a79-28ebe1c65913-multus-conf-dir\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.541324 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.483960 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/903e7441-02b7-49ee-a946-75875b1e34e3-registration-dir\") pod \"aws-ebs-csi-driver-node-9glkc\" (UID: \"903e7441-02b7-49ee-a946-75875b1e34e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glkc" Jul 08 19:03:39.541324 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:39.483983 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:03:39.541324 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.483995 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a1318ccd-51f3-4a63-9a79-28ebe1c65913-hostroot\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.543071 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.484020 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1318ccd-51f3-4a63-9a79-28ebe1c65913-cni-binary-copy\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.543071 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:39.484036 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99034b7e-72e2-4a28-8711-233630b7e9ea-metrics-certs podName:99034b7e-72e2-4a28-8711-233630b7e9ea nodeName:}" failed. No retries permitted until 2026-07-08 19:03:39.984019959 +0000 UTC m=+3.242972877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99034b7e-72e2-4a28-8711-233630b7e9ea-metrics-certs") pod "network-metrics-daemon-5rmts" (UID: "99034b7e-72e2-4a28-8711-233630b7e9ea") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:03:39.543071 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.484158 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/17f1251c-a915-41c2-a44a-fd7a048826af-konnectivity-ca\") pod \"konnectivity-agent-jwprt\" (UID: \"17f1251c-a915-41c2-a44a-fd7a048826af\") " pod="kube-system/konnectivity-agent-jwprt" Jul 08 19:03:39.543071 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.486786 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fc196626-817d-41e0-8bac-8c0ac61a559f-etc-tuned\") pod \"tuned-6mf6p\" (UID: \"fc196626-817d-41e0-8bac-8c0ac61a559f\") " pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" Jul 08 19:03:39.543071 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.487014 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/17f1251c-a915-41c2-a44a-fd7a048826af-agent-certs\") pod \"konnectivity-agent-jwprt\" (UID: \"17f1251c-a915-41c2-a44a-fd7a048826af\") " pod="kube-system/konnectivity-agent-jwprt" Jul 08 19:03:39.543071 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.488001 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc196626-817d-41e0-8bac-8c0ac61a559f-tmp\") pod \"tuned-6mf6p\" (UID: \"fc196626-817d-41e0-8bac-8c0ac61a559f\") " pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" Jul 08 19:03:39.543071 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.488076 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7873f48e-334b-4a54-8908-191e668b953d-ovn-node-metrics-cert\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.543071 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.493134 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 08 19:03:39.543071 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.493201 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 08 19:03:39.543071 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.493209 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 08 19:03:39.543071 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:39.493283 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 19:03:39.543071 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:39.493304 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 19:03:39.543071 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:39.493317 2558 projected.go:196] Error preparing data for projected volume kube-api-access-lszs7 for pod openshift-network-diagnostics/network-check-target-26v6h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:03:39.543071 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.493413 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 08 19:03:39.543071 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.493422 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2nhd7\"" Jul 08 19:03:39.543071 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:39.493447 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fcd111d-1192-4c3e-90e7-2294cdec33ee-kube-api-access-lszs7 podName:6fcd111d-1192-4c3e-90e7-2294cdec33ee nodeName:}" failed. No retries permitted until 2026-07-08 19:03:39.993430267 +0000 UTC m=+3.252383186 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lszs7" (UniqueName: "kubernetes.io/projected/6fcd111d-1192-4c3e-90e7-2294cdec33ee-kube-api-access-lszs7") pod "network-check-target-26v6h" (UID: "6fcd111d-1192-4c3e-90e7-2294cdec33ee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:03:39.543071 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.493665 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pdxs6\"" Jul 08 19:03:39.543903 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.493827 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9958d823-d456-441f-9bef-868175bf2db8-iptables-alerter-script\") pod \"iptables-alerter-xcqj8\" (UID: \"9958d823-d456-441f-9bef-868175bf2db8\") " pod="openshift-network-operator/iptables-alerter-xcqj8" Jul 08 19:03:39.543903 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.496167 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ps6w\" (UniqueName: \"kubernetes.io/projected/60145409-2aa4-4d68-9405-30b21fae674e-kube-api-access-6ps6w\") pod \"multus-additional-cni-plugins-7ks2q\" (UID: \"60145409-2aa4-4d68-9405-30b21fae674e\") " pod="openshift-multus/multus-additional-cni-plugins-7ks2q" Jul 08 19:03:39.543903 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.496554 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvmw8\" (UniqueName: \"kubernetes.io/projected/7873f48e-334b-4a54-8908-191e668b953d-kube-api-access-vvmw8\") pod \"ovnkube-node-6gmjt\" (UID: \"7873f48e-334b-4a54-8908-191e668b953d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.543903 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.498943 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grmf9\" (UniqueName: \"kubernetes.io/projected/74f5d0c6-337e-407c-bdef-75c0dcd3c47d-kube-api-access-grmf9\") pod \"node-resolver-44598\" (UID: \"74f5d0c6-337e-407c-bdef-75c0dcd3c47d\") " pod="openshift-dns/node-resolver-44598" Jul 08 19:03:39.543903 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.502944 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a1318ccd-51f3-4a63-9a79-28ebe1c65913-multus-daemon-config\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.567844 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.567752 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9rp4\" (UniqueName: \"kubernetes.io/projected/a0658e5c-4e22-43d3-ac11-86df5ff39969-kube-api-access-r9rp4\") pod \"node-ca-gktll\" (UID: \"a0658e5c-4e22-43d3-ac11-86df5ff39969\") " pod="openshift-image-registry/node-ca-gktll" Jul 08 19:03:39.570517 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.570469 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2px8t\" (UniqueName: \"kubernetes.io/projected/fc196626-817d-41e0-8bac-8c0ac61a559f-kube-api-access-2px8t\") pod \"tuned-6mf6p\" (UID: \"fc196626-817d-41e0-8bac-8c0ac61a559f\") " pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" Jul 08 19:03:39.570517 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.570508 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n5bt\" (UniqueName: \"kubernetes.io/projected/903e7441-02b7-49ee-a946-75875b1e34e3-kube-api-access-4n5bt\") pod \"aws-ebs-csi-driver-node-9glkc\" (UID: \"903e7441-02b7-49ee-a946-75875b1e34e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glkc" Jul 08 19:03:39.571069 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.571047 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pd2g\" (UniqueName: \"kubernetes.io/projected/a1318ccd-51f3-4a63-9a79-28ebe1c65913-kube-api-access-8pd2g\") pod \"multus-rb7nx\" (UID: \"a1318ccd-51f3-4a63-9a79-28ebe1c65913\") " pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.571241 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.571217 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mctt2\" (UniqueName: \"kubernetes.io/projected/9958d823-d456-441f-9bef-868175bf2db8-kube-api-access-mctt2\") pod \"iptables-alerter-xcqj8\" (UID: \"9958d823-d456-441f-9bef-868175bf2db8\") " pod="openshift-network-operator/iptables-alerter-xcqj8" Jul 08 19:03:39.571513 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.571483 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd5c6\" (UniqueName: \"kubernetes.io/projected/99034b7e-72e2-4a28-8711-233630b7e9ea-kube-api-access-zd5c6\") pod \"network-metrics-daemon-5rmts\" (UID: \"99034b7e-72e2-4a28-8711-233630b7e9ea\") " pod="openshift-multus/network-metrics-daemon-5rmts" Jul 08 19:03:39.589295 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.589273 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7ks2q" Jul 08 19:03:39.597747 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.597717 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jwprt" Jul 08 19:03:39.602794 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:39.602763 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60145409_2aa4_4d68_9405_30b21fae674e.slice/crio-5835a0bb54b45aa15360a3f3a8119db81af214d431e18a46cdcde9c13350d60b WatchSource:0}: Error finding container 5835a0bb54b45aa15360a3f3a8119db81af214d431e18a46cdcde9c13350d60b: Status 404 returned error can't find the container with id 5835a0bb54b45aa15360a3f3a8119db81af214d431e18a46cdcde9c13350d60b Jul 08 19:03:39.603851 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.603829 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glkc" Jul 08 19:03:39.607982 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:39.607958 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17f1251c_a915_41c2_a44a_fd7a048826af.slice/crio-519576a0a82354b65c4e0e899abf89627a50be7f0beb6137301ecc632fcb8c87 WatchSource:0}: Error finding container 519576a0a82354b65c4e0e899abf89627a50be7f0beb6137301ecc632fcb8c87: Status 404 returned error can't find the container with id 519576a0a82354b65c4e0e899abf89627a50be7f0beb6137301ecc632fcb8c87 Jul 08 19:03:39.613113 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.613091 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" Jul 08 19:03:39.614242 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:39.614215 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod903e7441_02b7_49ee_a946_75875b1e34e3.slice/crio-cad8d1e4cfcbc98f5aec391525e2c9b8b8044ad2ce6ce38a2228234e80a1b8bc WatchSource:0}: Error finding container cad8d1e4cfcbc98f5aec391525e2c9b8b8044ad2ce6ce38a2228234e80a1b8bc: Status 404 returned error can't find the container with id cad8d1e4cfcbc98f5aec391525e2c9b8b8044ad2ce6ce38a2228234e80a1b8bc Jul 08 19:03:39.623399 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:39.623369 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc196626_817d_41e0_8bac_8c0ac61a559f.slice/crio-55f69582b33208abc957cc0404f17a250be21996864940baaf8750b25237c8ed WatchSource:0}: Error finding container 55f69582b33208abc957cc0404f17a250be21996864940baaf8750b25237c8ed: Status 404 returned error can't find the container with id 55f69582b33208abc957cc0404f17a250be21996864940baaf8750b25237c8ed Jul 08 19:03:39.772741 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.772715 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:03:39.779319 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.779236 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-44598" Jul 08 19:03:39.779319 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.779255 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xcqj8" Jul 08 19:03:39.786512 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.786493 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gktll" Jul 08 19:03:39.806084 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.806065 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rb7nx" Jul 08 19:03:39.987232 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.987140 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/640044e9-7c83-4f5a-9c9d-2e4183284806-original-pull-secret\") pod \"global-pull-secret-syncer-mlw2c\" (UID: \"640044e9-7c83-4f5a-9c9d-2e4183284806\") " pod="kube-system/global-pull-secret-syncer-mlw2c" Jul 08 19:03:39.987232 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:39.987212 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99034b7e-72e2-4a28-8711-233630b7e9ea-metrics-certs\") pod \"network-metrics-daemon-5rmts\" (UID: \"99034b7e-72e2-4a28-8711-233630b7e9ea\") " pod="openshift-multus/network-metrics-daemon-5rmts" Jul 08 19:03:39.987441 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:39.987301 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 19:03:39.987441 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:39.987376 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/640044e9-7c83-4f5a-9c9d-2e4183284806-original-pull-secret podName:640044e9-7c83-4f5a-9c9d-2e4183284806 nodeName:}" failed. No retries permitted until 2026-07-08 19:03:40.987358358 +0000 UTC m=+4.246311260 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/640044e9-7c83-4f5a-9c9d-2e4183284806-original-pull-secret") pod "global-pull-secret-syncer-mlw2c" (UID: "640044e9-7c83-4f5a-9c9d-2e4183284806") : object "kube-system"/"original-pull-secret" not registered Jul 08 19:03:39.987441 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:39.987305 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:03:39.987639 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:39.987446 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99034b7e-72e2-4a28-8711-233630b7e9ea-metrics-certs podName:99034b7e-72e2-4a28-8711-233630b7e9ea nodeName:}" failed. No retries permitted until 2026-07-08 19:03:40.987429496 +0000 UTC m=+4.246382399 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99034b7e-72e2-4a28-8711-233630b7e9ea-metrics-certs") pod "network-metrics-daemon-5rmts" (UID: "99034b7e-72e2-4a28-8711-233630b7e9ea") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:03:40.088431 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:40.088401 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lszs7\" (UniqueName: \"kubernetes.io/projected/6fcd111d-1192-4c3e-90e7-2294cdec33ee-kube-api-access-lszs7\") pod \"network-check-target-26v6h\" (UID: \"6fcd111d-1192-4c3e-90e7-2294cdec33ee\") " pod="openshift-network-diagnostics/network-check-target-26v6h" Jul 08 19:03:40.088571 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:40.088526 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 19:03:40.088571 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:40.088546 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 19:03:40.088571 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:40.088555 2558 projected.go:196] Error preparing data for projected volume kube-api-access-lszs7 for pod openshift-network-diagnostics/network-check-target-26v6h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:03:40.088699 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:40.088620 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fcd111d-1192-4c3e-90e7-2294cdec33ee-kube-api-access-lszs7 podName:6fcd111d-1192-4c3e-90e7-2294cdec33ee nodeName:}" failed. No retries permitted until 2026-07-08 19:03:41.088606851 +0000 UTC m=+4.347559754 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lszs7" (UniqueName: "kubernetes.io/projected/6fcd111d-1192-4c3e-90e7-2294cdec33ee-kube-api-access-lszs7") pod "network-check-target-26v6h" (UID: "6fcd111d-1192-4c3e-90e7-2294cdec33ee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:03:40.275693 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:40.275624 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26v6h" Jul 08 19:03:40.276124 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:40.275727 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26v6h" podUID="6fcd111d-1192-4c3e-90e7-2294cdec33ee" Jul 08 19:03:40.287801 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:40.287761 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glkc" event={"ID":"903e7441-02b7-49ee-a946-75875b1e34e3","Type":"ContainerStarted","Data":"cad8d1e4cfcbc98f5aec391525e2c9b8b8044ad2ce6ce38a2228234e80a1b8bc"} Jul 08 19:03:40.288918 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:40.288891 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jwprt" event={"ID":"17f1251c-a915-41c2-a44a-fd7a048826af","Type":"ContainerStarted","Data":"519576a0a82354b65c4e0e899abf89627a50be7f0beb6137301ecc632fcb8c87"} Jul 08 19:03:40.290010 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:40.289986 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7ks2q" event={"ID":"60145409-2aa4-4d68-9405-30b21fae674e","Type":"ContainerStarted","Data":"5835a0bb54b45aa15360a3f3a8119db81af214d431e18a46cdcde9c13350d60b"} Jul 08 19:03:40.291133 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:40.291111 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" event={"ID":"fc196626-817d-41e0-8bac-8c0ac61a559f","Type":"ContainerStarted","Data":"55f69582b33208abc957cc0404f17a250be21996864940baaf8750b25237c8ed"} Jul 08 19:03:40.483030 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:40.482957 2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 18:58:39 +0000 UTC" deadline="2028-03-14 16:59:42.607736362 +0000 UTC" Jul 08 19:03:40.483030 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:40.483026 2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14757h56m2.124713916s" Jul 08 19:03:40.770045 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:40.769936 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0658e5c_4e22_43d3_ac11_86df5ff39969.slice/crio-a9e32e6720b089428a7ed189459bd6e149ad91a5379674cde1e72e3b3373ef03 WatchSource:0}: Error finding container a9e32e6720b089428a7ed189459bd6e149ad91a5379674cde1e72e3b3373ef03: Status 404 returned error can't find the container with id a9e32e6720b089428a7ed189459bd6e149ad91a5379674cde1e72e3b3373ef03 Jul 08 19:03:40.771870 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:40.771851 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7873f48e_334b_4a54_8908_191e668b953d.slice/crio-a345c66c292708c37991247979e6af84254e61a397b9c7f0e58261f1ebabd0c0 WatchSource:0}: Error finding container a345c66c292708c37991247979e6af84254e61a397b9c7f0e58261f1ebabd0c0: Status 404 returned error can't find the container with id a345c66c292708c37991247979e6af84254e61a397b9c7f0e58261f1ebabd0c0 Jul 08 19:03:40.773872 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:40.773849 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74f5d0c6_337e_407c_bdef_75c0dcd3c47d.slice/crio-47782e9034530e912ba81140c09dc6e6b90830e601d662f40e6c3b44fcf3e608 WatchSource:0}: Error finding container 47782e9034530e912ba81140c09dc6e6b90830e601d662f40e6c3b44fcf3e608: Status 404 returned error can't find the container with id 47782e9034530e912ba81140c09dc6e6b90830e601d662f40e6c3b44fcf3e608 Jul 08 19:03:40.775082 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:40.775049 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1318ccd_51f3_4a63_9a79_28ebe1c65913.slice/crio-c175ae25d18897ec2ef50fc68555ba70ec31a168bfa3f66a71ba0572781eb115 WatchSource:0}: Error finding container c175ae25d18897ec2ef50fc68555ba70ec31a168bfa3f66a71ba0572781eb115: Status 404 returned error can't find the container with id c175ae25d18897ec2ef50fc68555ba70ec31a168bfa3f66a71ba0572781eb115 Jul 08 19:03:40.776152 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:03:40.776107 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9958d823_d456_441f_9bef_868175bf2db8.slice/crio-8e8184eca53a7422e86ae97b8647db05c813cfddb75082c1fa38fb60cf478e79 WatchSource:0}: Error finding container 8e8184eca53a7422e86ae97b8647db05c813cfddb75082c1fa38fb60cf478e79: Status 404 returned error can't find the container with id 8e8184eca53a7422e86ae97b8647db05c813cfddb75082c1fa38fb60cf478e79 Jul 08 19:03:40.994888 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:40.994615 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/640044e9-7c83-4f5a-9c9d-2e4183284806-original-pull-secret\") pod \"global-pull-secret-syncer-mlw2c\" (UID: \"640044e9-7c83-4f5a-9c9d-2e4183284806\") " pod="kube-system/global-pull-secret-syncer-mlw2c" Jul 08 19:03:40.995040 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:40.994952 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99034b7e-72e2-4a28-8711-233630b7e9ea-metrics-certs\") pod \"network-metrics-daemon-5rmts\" (UID: \"99034b7e-72e2-4a28-8711-233630b7e9ea\") " pod="openshift-multus/network-metrics-daemon-5rmts" Jul 08 19:03:40.995040 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:40.994788 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 19:03:40.995129 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:40.995048 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/640044e9-7c83-4f5a-9c9d-2e4183284806-original-pull-secret podName:640044e9-7c83-4f5a-9c9d-2e4183284806 nodeName:}" failed. No retries permitted until 2026-07-08 19:03:42.995015882 +0000 UTC m=+6.253968787 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/640044e9-7c83-4f5a-9c9d-2e4183284806-original-pull-secret") pod "global-pull-secret-syncer-mlw2c" (UID: "640044e9-7c83-4f5a-9c9d-2e4183284806") : object "kube-system"/"original-pull-secret" not registered Jul 08 19:03:40.995129 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:40.995091 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:03:40.995207 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:40.995148 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99034b7e-72e2-4a28-8711-233630b7e9ea-metrics-certs podName:99034b7e-72e2-4a28-8711-233630b7e9ea nodeName:}" failed. No retries permitted until 2026-07-08 19:03:42.995132719 +0000 UTC m=+6.254085625 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99034b7e-72e2-4a28-8711-233630b7e9ea-metrics-certs") pod "network-metrics-daemon-5rmts" (UID: "99034b7e-72e2-4a28-8711-233630b7e9ea") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:03:41.097426 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:41.096812 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lszs7\" (UniqueName: \"kubernetes.io/projected/6fcd111d-1192-4c3e-90e7-2294cdec33ee-kube-api-access-lszs7\") pod \"network-check-target-26v6h\" (UID: \"6fcd111d-1192-4c3e-90e7-2294cdec33ee\") " pod="openshift-network-diagnostics/network-check-target-26v6h" Jul 08 19:03:41.097426 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:41.097023 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 19:03:41.097426 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:41.097045 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 19:03:41.097426 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:41.097057 2558 projected.go:196] Error preparing data for projected volume kube-api-access-lszs7 for pod openshift-network-diagnostics/network-check-target-26v6h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:03:41.097426 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:41.097110 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fcd111d-1192-4c3e-90e7-2294cdec33ee-kube-api-access-lszs7 podName:6fcd111d-1192-4c3e-90e7-2294cdec33ee nodeName:}" failed. No retries permitted until 2026-07-08 19:03:43.097093267 +0000 UTC m=+6.356046174 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lszs7" (UniqueName: "kubernetes.io/projected/6fcd111d-1192-4c3e-90e7-2294cdec33ee-kube-api-access-lszs7") pod "network-check-target-26v6h" (UID: "6fcd111d-1192-4c3e-90e7-2294cdec33ee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:03:41.278328 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:41.277689 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5rmts" Jul 08 19:03:41.278328 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:41.277820 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5rmts" podUID="99034b7e-72e2-4a28-8711-233630b7e9ea" Jul 08 19:03:41.278328 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:41.278149 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlw2c" Jul 08 19:03:41.278328 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:41.278243 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlw2c" podUID="640044e9-7c83-4f5a-9c9d-2e4183284806" Jul 08 19:03:41.294519 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:41.294490 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rb7nx" event={"ID":"a1318ccd-51f3-4a63-9a79-28ebe1c65913","Type":"ContainerStarted","Data":"c175ae25d18897ec2ef50fc68555ba70ec31a168bfa3f66a71ba0572781eb115"} Jul 08 19:03:41.296221 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:41.296193 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" event={"ID":"7873f48e-334b-4a54-8908-191e668b953d","Type":"ContainerStarted","Data":"a345c66c292708c37991247979e6af84254e61a397b9c7f0e58261f1ebabd0c0"} Jul 08 19:03:41.298302 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:41.298274 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gktll" event={"ID":"a0658e5c-4e22-43d3-ac11-86df5ff39969","Type":"ContainerStarted","Data":"a9e32e6720b089428a7ed189459bd6e149ad91a5379674cde1e72e3b3373ef03"} Jul 08 19:03:41.301057 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:41.301030 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-191.ec2.internal" event={"ID":"ce1e796271b5d048919b6c245c7db2fc","Type":"ContainerStarted","Data":"65a673a2ba21125369c3ae03e1fa42c152b748feeb1a050279290e2bf56b8e84"} Jul 08 19:03:41.304634 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:41.304049 2558 generic.go:356] "Generic (PLEG): container finished" podID="330f763c6aa8f4d35b771e6847835d74" containerID="1d36c0a5c395c3f00cbf8236542d91c9ff16aa3570c6db19c5288705d790a5e5" exitCode=0 Jul 08 19:03:41.304634 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:41.304132 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-191.ec2.internal" event={"ID":"330f763c6aa8f4d35b771e6847835d74","Type":"ContainerDied","Data":"1d36c0a5c395c3f00cbf8236542d91c9ff16aa3570c6db19c5288705d790a5e5"} Jul 08 19:03:41.307805 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:41.307782 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xcqj8" event={"ID":"9958d823-d456-441f-9bef-868175bf2db8","Type":"ContainerStarted","Data":"8e8184eca53a7422e86ae97b8647db05c813cfddb75082c1fa38fb60cf478e79"} Jul 08 19:03:41.309155 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:41.309130 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-44598" event={"ID":"74f5d0c6-337e-407c-bdef-75c0dcd3c47d","Type":"ContainerStarted","Data":"47782e9034530e912ba81140c09dc6e6b90830e601d662f40e6c3b44fcf3e608"} Jul 08 19:03:41.320328 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:41.319790 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-191.ec2.internal" podStartSLOduration=3.319774293 podStartE2EDuration="3.319774293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:03:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:03:41.319431247 +0000 UTC m=+4.578384173" watchObservedRunningTime="2026-07-08 19:03:41.319774293 +0000 UTC m=+4.578727219" Jul 08 19:03:41.488091 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:41.484046 2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 18:58:39 +0000 UTC" deadline="2028-03-14 04:50:37.231549537 +0000 UTC" Jul 08 19:03:41.488091 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:41.484080 2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14745h46m55.747473669s" Jul 08 19:03:42.276426 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:42.276392 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26v6h" Jul 08 19:03:42.276628 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:42.276534 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26v6h" podUID="6fcd111d-1192-4c3e-90e7-2294cdec33ee" Jul 08 19:03:42.336413 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:42.336376 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-191.ec2.internal" event={"ID":"330f763c6aa8f4d35b771e6847835d74","Type":"ContainerStarted","Data":"f295bccfd7afcb3094f3b7024a34fe8c3b249da43f1da4a8ffd850ce85df7894"} Jul 08 19:03:43.011505 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:43.011467 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/640044e9-7c83-4f5a-9c9d-2e4183284806-original-pull-secret\") pod \"global-pull-secret-syncer-mlw2c\" (UID: \"640044e9-7c83-4f5a-9c9d-2e4183284806\") " pod="kube-system/global-pull-secret-syncer-mlw2c" Jul 08 19:03:43.011729 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:43.011557 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99034b7e-72e2-4a28-8711-233630b7e9ea-metrics-certs\") pod \"network-metrics-daemon-5rmts\" (UID: \"99034b7e-72e2-4a28-8711-233630b7e9ea\") " pod="openshift-multus/network-metrics-daemon-5rmts" Jul 08 19:03:43.011798 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:43.011743 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:03:43.011850 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:43.011804 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99034b7e-72e2-4a28-8711-233630b7e9ea-metrics-certs podName:99034b7e-72e2-4a28-8711-233630b7e9ea nodeName:}" failed. No retries permitted until 2026-07-08 19:03:47.011787181 +0000 UTC m=+10.270740087 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99034b7e-72e2-4a28-8711-233630b7e9ea-metrics-certs") pod "network-metrics-daemon-5rmts" (UID: "99034b7e-72e2-4a28-8711-233630b7e9ea") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:03:43.012228 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:43.012206 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 19:03:43.012303 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:43.012265 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/640044e9-7c83-4f5a-9c9d-2e4183284806-original-pull-secret podName:640044e9-7c83-4f5a-9c9d-2e4183284806 nodeName:}" failed. No retries permitted until 2026-07-08 19:03:47.012249244 +0000 UTC m=+10.271202170 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/640044e9-7c83-4f5a-9c9d-2e4183284806-original-pull-secret") pod "global-pull-secret-syncer-mlw2c" (UID: "640044e9-7c83-4f5a-9c9d-2e4183284806") : object "kube-system"/"original-pull-secret" not registered Jul 08 19:03:43.112637 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:43.112566 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lszs7\" (UniqueName: \"kubernetes.io/projected/6fcd111d-1192-4c3e-90e7-2294cdec33ee-kube-api-access-lszs7\") pod \"network-check-target-26v6h\" (UID: \"6fcd111d-1192-4c3e-90e7-2294cdec33ee\") " pod="openshift-network-diagnostics/network-check-target-26v6h" Jul 08 19:03:43.112801 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:43.112786 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 19:03:43.112873 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:43.112806 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 19:03:43.112873 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:43.112844 2558 projected.go:196] Error preparing data for projected volume kube-api-access-lszs7 for pod openshift-network-diagnostics/network-check-target-26v6h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:03:43.112965 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:43.112906 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fcd111d-1192-4c3e-90e7-2294cdec33ee-kube-api-access-lszs7 podName:6fcd111d-1192-4c3e-90e7-2294cdec33ee nodeName:}" failed. No retries permitted until 2026-07-08 19:03:47.112885496 +0000 UTC m=+10.371838422 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lszs7" (UniqueName: "kubernetes.io/projected/6fcd111d-1192-4c3e-90e7-2294cdec33ee-kube-api-access-lszs7") pod "network-check-target-26v6h" (UID: "6fcd111d-1192-4c3e-90e7-2294cdec33ee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:03:43.277269 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:43.276686 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5rmts" Jul 08 19:03:43.277269 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:43.276840 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5rmts" podUID="99034b7e-72e2-4a28-8711-233630b7e9ea" Jul 08 19:03:43.277742 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:43.277572 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlw2c" Jul 08 19:03:43.277742 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:43.277693 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlw2c" podUID="640044e9-7c83-4f5a-9c9d-2e4183284806" Jul 08 19:03:44.275785 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:44.275747 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26v6h" Jul 08 19:03:44.276257 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:44.275874 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26v6h" podUID="6fcd111d-1192-4c3e-90e7-2294cdec33ee" Jul 08 19:03:45.275915 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:45.275882 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5rmts" Jul 08 19:03:45.276354 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:45.276111 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5rmts" podUID="99034b7e-72e2-4a28-8711-233630b7e9ea" Jul 08 19:03:45.276593 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:45.276556 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlw2c" Jul 08 19:03:45.276700 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:45.276677 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlw2c" podUID="640044e9-7c83-4f5a-9c9d-2e4183284806" Jul 08 19:03:46.275750 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:46.275723 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26v6h" Jul 08 19:03:46.275913 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:46.275827 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26v6h" podUID="6fcd111d-1192-4c3e-90e7-2294cdec33ee" Jul 08 19:03:47.043540 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:47.043491 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/640044e9-7c83-4f5a-9c9d-2e4183284806-original-pull-secret\") pod \"global-pull-secret-syncer-mlw2c\" (UID: \"640044e9-7c83-4f5a-9c9d-2e4183284806\") " pod="kube-system/global-pull-secret-syncer-mlw2c" Jul 08 19:03:47.044054 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:47.043604 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99034b7e-72e2-4a28-8711-233630b7e9ea-metrics-certs\") pod \"network-metrics-daemon-5rmts\" (UID: \"99034b7e-72e2-4a28-8711-233630b7e9ea\") " pod="openshift-multus/network-metrics-daemon-5rmts" Jul 08 19:03:47.044054 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:47.043715 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 19:03:47.044054 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:47.043746 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:03:47.044054 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:47.043807 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99034b7e-72e2-4a28-8711-233630b7e9ea-metrics-certs podName:99034b7e-72e2-4a28-8711-233630b7e9ea nodeName:}" failed. No retries permitted until 2026-07-08 19:03:55.043786066 +0000 UTC m=+18.302738970 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99034b7e-72e2-4a28-8711-233630b7e9ea-metrics-certs") pod "network-metrics-daemon-5rmts" (UID: "99034b7e-72e2-4a28-8711-233630b7e9ea") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:03:47.044054 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:47.043827 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/640044e9-7c83-4f5a-9c9d-2e4183284806-original-pull-secret podName:640044e9-7c83-4f5a-9c9d-2e4183284806 nodeName:}" failed. No retries permitted until 2026-07-08 19:03:55.043817315 +0000 UTC m=+18.302770222 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/640044e9-7c83-4f5a-9c9d-2e4183284806-original-pull-secret") pod "global-pull-secret-syncer-mlw2c" (UID: "640044e9-7c83-4f5a-9c9d-2e4183284806") : object "kube-system"/"original-pull-secret" not registered Jul 08 19:03:47.144991 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:47.144931 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lszs7\" (UniqueName: \"kubernetes.io/projected/6fcd111d-1192-4c3e-90e7-2294cdec33ee-kube-api-access-lszs7\") pod \"network-check-target-26v6h\" (UID: \"6fcd111d-1192-4c3e-90e7-2294cdec33ee\") " pod="openshift-network-diagnostics/network-check-target-26v6h" Jul 08 19:03:47.145182 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:47.145120 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 19:03:47.145182 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:47.145141 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 19:03:47.145182 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:47.145152 2558 projected.go:196] Error preparing data for projected volume kube-api-access-lszs7 for pod openshift-network-diagnostics/network-check-target-26v6h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:03:47.145328 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:47.145213 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fcd111d-1192-4c3e-90e7-2294cdec33ee-kube-api-access-lszs7 podName:6fcd111d-1192-4c3e-90e7-2294cdec33ee nodeName:}" failed. No retries permitted until 2026-07-08 19:03:55.145194779 +0000 UTC m=+18.404147697 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lszs7" (UniqueName: "kubernetes.io/projected/6fcd111d-1192-4c3e-90e7-2294cdec33ee-kube-api-access-lszs7") pod "network-check-target-26v6h" (UID: "6fcd111d-1192-4c3e-90e7-2294cdec33ee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:03:47.276639 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:47.276607 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5rmts" Jul 08 19:03:47.276831 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:47.276739 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5rmts" podUID="99034b7e-72e2-4a28-8711-233630b7e9ea" Jul 08 19:03:47.277245 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:47.277042 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlw2c" Jul 08 19:03:47.277245 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:47.277133 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlw2c" podUID="640044e9-7c83-4f5a-9c9d-2e4183284806" Jul 08 19:03:48.276449 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:48.276371 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26v6h" Jul 08 19:03:48.276869 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:48.276490 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26v6h" podUID="6fcd111d-1192-4c3e-90e7-2294cdec33ee" Jul 08 19:03:49.276282 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:49.276248 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5rmts" Jul 08 19:03:49.276282 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:49.276272 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlw2c" Jul 08 19:03:49.276507 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:49.276397 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5rmts" podUID="99034b7e-72e2-4a28-8711-233630b7e9ea" Jul 08 19:03:49.276858 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:49.276539 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlw2c" podUID="640044e9-7c83-4f5a-9c9d-2e4183284806" Jul 08 19:03:50.276055 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:50.276021 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26v6h" Jul 08 19:03:50.276229 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:50.276149 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26v6h" podUID="6fcd111d-1192-4c3e-90e7-2294cdec33ee" Jul 08 19:03:51.275878 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:51.275649 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5rmts" Jul 08 19:03:51.276308 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:51.275699 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlw2c" Jul 08 19:03:51.276308 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:51.275910 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5rmts" podUID="99034b7e-72e2-4a28-8711-233630b7e9ea" Jul 08 19:03:51.276308 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:51.275956 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlw2c" podUID="640044e9-7c83-4f5a-9c9d-2e4183284806" Jul 08 19:03:51.357988 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:51.357948 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" event={"ID":"fc196626-817d-41e0-8bac-8c0ac61a559f","Type":"ContainerStarted","Data":"620db60aaf4daf787f77e989f6930753bc6610a4cfc8d0d8834e023de8298261"} Jul 08 19:03:51.360273 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:51.360235 2558 generic.go:356] "Generic (PLEG): container finished" podID="60145409-2aa4-4d68-9405-30b21fae674e" containerID="8c3a1fa94c48558830822040c091d92b8a00bfaf4fd867b039d05d0b2a909140" exitCode=0 Jul 08 19:03:51.360873 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:51.360846 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7ks2q" event={"ID":"60145409-2aa4-4d68-9405-30b21fae674e","Type":"ContainerDied","Data":"8c3a1fa94c48558830822040c091d92b8a00bfaf4fd867b039d05d0b2a909140"} Jul 08 19:03:51.363792 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:51.363765 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-44598" event={"ID":"74f5d0c6-337e-407c-bdef-75c0dcd3c47d","Type":"ContainerStarted","Data":"6884342f5d1602b8bb87aa2299219bb79cd4929bf680acd4021eb52b63a240b8"} Jul 08 19:03:51.365644 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:51.365624 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glkc" event={"ID":"903e7441-02b7-49ee-a946-75875b1e34e3","Type":"ContainerStarted","Data":"2d62127b83c8c8c9eaafa942a2f304a67afe01533ca7456579031687aa72876e"} Jul 08 19:03:51.367374 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:51.367324 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jwprt" event={"ID":"17f1251c-a915-41c2-a44a-fd7a048826af","Type":"ContainerStarted","Data":"efb9ac0a1ab42db12e60d3d7c9068cd3cfedc41552083bae9f75af6b6b06c7cf"} Jul 08 19:03:51.369431 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:51.369412 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gktll" event={"ID":"a0658e5c-4e22-43d3-ac11-86df5ff39969","Type":"ContainerStarted","Data":"58403b9626d92a24d0f0a237c05a0d89042e0356837196694524ca25a7f2f943"} Jul 08 19:03:51.376806 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:51.376742 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6mf6p" podStartSLOduration=3.356290249 podStartE2EDuration="14.376705969s" totalImagesPullingTime="11.02041572s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:03:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:03:51.37651764 +0000 UTC m=+14.635470566" watchObservedRunningTime="2026-07-08 19:03:51.376705969 +0000 UTC m=+14.635658895" Jul 08 19:03:51.377403 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:51.377367 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-191.ec2.internal" podStartSLOduration=13.377352939 podStartE2EDuration="13.377352939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:03:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:03:42.357819037 +0000 UTC m=+5.616771964" watchObservedRunningTime="2026-07-08 19:03:51.377352939 +0000 UTC m=+14.636305865" Jul 08 19:03:51.391617 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:51.391502 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-44598" podStartSLOduration=4.816836014 podStartE2EDuration="14.391487431s" totalImagesPullingTime="9.574651417s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:03:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:03:51.391404942 +0000 UTC m=+14.650357867" watchObservedRunningTime="2026-07-08 19:03:51.391487431 +0000 UTC m=+14.650440357" Jul 08 19:03:51.416279 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:51.416215 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jwprt" podStartSLOduration=3.7292173589999997 podStartE2EDuration="14.416200255s" totalImagesPullingTime="10.686982896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:03:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:03:51.416103105 +0000 UTC m=+14.675056031" watchObservedRunningTime="2026-07-08 19:03:51.416200255 +0000 UTC m=+14.675153181" Jul 08 19:03:51.432147 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:51.432104 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gktll" podStartSLOduration=4.858625273 podStartE2EDuration="14.432095189s" totalImagesPullingTime="9.573469916s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:03:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:03:51.432090711 +0000 UTC m=+14.691043637" watchObservedRunningTime="2026-07-08 19:03:51.432095189 +0000 UTC m=+14.691048116" Jul 08 19:03:52.275970 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:52.275938 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26v6h" Jul 08 19:03:52.276381 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:52.276048 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26v6h" podUID="6fcd111d-1192-4c3e-90e7-2294cdec33ee" Jul 08 19:03:52.322245 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:52.322210 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jwprt" Jul 08 19:03:52.323156 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:52.323133 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jwprt" Jul 08 19:03:52.373746 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:52.373705 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xcqj8" event={"ID":"9958d823-d456-441f-9bef-868175bf2db8","Type":"ContainerStarted","Data":"966f06fb32025f43d37437fddee9ada4bcf1145358ed6a265ce4224e1a54b80a"} Jul 08 19:03:52.407740 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:52.407680 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xcqj8" podStartSLOduration=5.888889531 podStartE2EDuration="15.40765516s" totalImagesPullingTime="9.518765629s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:03:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:03:52.407383381 +0000 UTC m=+15.666336307" watchObservedRunningTime="2026-07-08 19:03:52.40765516 +0000 UTC m=+15.666608081" Jul 08 19:03:53.275748 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:53.275715 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5rmts" Jul 08 19:03:53.275900 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:53.275848 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5rmts" podUID="99034b7e-72e2-4a28-8711-233630b7e9ea" Jul 08 19:03:53.276016 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:53.275998 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlw2c" Jul 08 19:03:53.276412 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:53.276092 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlw2c" podUID="640044e9-7c83-4f5a-9c9d-2e4183284806" Jul 08 19:03:53.375373 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:53.375344 2558 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 08 19:03:54.276285 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:54.276250 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26v6h" Jul 08 19:03:54.276921 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:54.276374 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26v6h" podUID="6fcd111d-1192-4c3e-90e7-2294cdec33ee" Jul 08 19:03:55.104093 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:55.104047 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99034b7e-72e2-4a28-8711-233630b7e9ea-metrics-certs\") pod \"network-metrics-daemon-5rmts\" (UID: \"99034b7e-72e2-4a28-8711-233630b7e9ea\") " pod="openshift-multus/network-metrics-daemon-5rmts" Jul 08 19:03:55.104446 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:55.104120 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/640044e9-7c83-4f5a-9c9d-2e4183284806-original-pull-secret\") pod \"global-pull-secret-syncer-mlw2c\" (UID: \"640044e9-7c83-4f5a-9c9d-2e4183284806\") " pod="kube-system/global-pull-secret-syncer-mlw2c" Jul 08 19:03:55.104446 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:55.104204 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:03:55.104446 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:55.104223 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 19:03:55.104446 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:55.104267 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99034b7e-72e2-4a28-8711-233630b7e9ea-metrics-certs podName:99034b7e-72e2-4a28-8711-233630b7e9ea nodeName:}" failed. No retries permitted until 2026-07-08 19:04:11.104246007 +0000 UTC m=+34.363198913 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99034b7e-72e2-4a28-8711-233630b7e9ea-metrics-certs") pod "network-metrics-daemon-5rmts" (UID: "99034b7e-72e2-4a28-8711-233630b7e9ea") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:03:55.104446 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:55.104286 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/640044e9-7c83-4f5a-9c9d-2e4183284806-original-pull-secret podName:640044e9-7c83-4f5a-9c9d-2e4183284806 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:11.104276276 +0000 UTC m=+34.363229185 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/640044e9-7c83-4f5a-9c9d-2e4183284806-original-pull-secret") pod "global-pull-secret-syncer-mlw2c" (UID: "640044e9-7c83-4f5a-9c9d-2e4183284806") : object "kube-system"/"original-pull-secret" not registered Jul 08 19:03:55.205487 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:55.205452 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lszs7\" (UniqueName: \"kubernetes.io/projected/6fcd111d-1192-4c3e-90e7-2294cdec33ee-kube-api-access-lszs7\") pod \"network-check-target-26v6h\" (UID: \"6fcd111d-1192-4c3e-90e7-2294cdec33ee\") " pod="openshift-network-diagnostics/network-check-target-26v6h" Jul 08 19:03:55.205699 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:55.205668 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 19:03:55.205699 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:55.205691 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 19:03:55.205790 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:55.205703 2558 projected.go:196] Error preparing data for projected volume kube-api-access-lszs7 for pod openshift-network-diagnostics/network-check-target-26v6h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:03:55.205790 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:55.205767 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fcd111d-1192-4c3e-90e7-2294cdec33ee-kube-api-access-lszs7 podName:6fcd111d-1192-4c3e-90e7-2294cdec33ee nodeName:}" failed. No retries permitted until 2026-07-08 19:04:11.205747181 +0000 UTC m=+34.464700087 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lszs7" (UniqueName: "kubernetes.io/projected/6fcd111d-1192-4c3e-90e7-2294cdec33ee-kube-api-access-lszs7") pod "network-check-target-26v6h" (UID: "6fcd111d-1192-4c3e-90e7-2294cdec33ee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:03:55.276458 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:55.276397 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5rmts" Jul 08 19:03:55.277383 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:55.276536 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5rmts" podUID="99034b7e-72e2-4a28-8711-233630b7e9ea" Jul 08 19:03:55.277383 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:55.276977 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlw2c" Jul 08 19:03:55.277383 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:55.277048 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlw2c" podUID="640044e9-7c83-4f5a-9c9d-2e4183284806" Jul 08 19:03:56.276523 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:56.276485 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26v6h" Jul 08 19:03:56.277027 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:56.276668 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26v6h" podUID="6fcd111d-1192-4c3e-90e7-2294cdec33ee" Jul 08 19:03:57.277194 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:57.277168 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlw2c" Jul 08 19:03:57.277645 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:57.277259 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlw2c" podUID="640044e9-7c83-4f5a-9c9d-2e4183284806" Jul 08 19:03:57.277645 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:57.277362 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5rmts" Jul 08 19:03:57.277645 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:57.277474 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5rmts" podUID="99034b7e-72e2-4a28-8711-233630b7e9ea" Jul 08 19:03:58.276309 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:58.276280 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26v6h" Jul 08 19:03:58.276482 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:58.276401 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26v6h" podUID="6fcd111d-1192-4c3e-90e7-2294cdec33ee" Jul 08 19:03:59.275828 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:59.275794 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5rmts" Jul 08 19:03:59.276260 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:03:59.275799 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlw2c" Jul 08 19:03:59.276260 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:59.275927 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5rmts" podUID="99034b7e-72e2-4a28-8711-233630b7e9ea" Jul 08 19:03:59.276260 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:03:59.276013 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlw2c" podUID="640044e9-7c83-4f5a-9c9d-2e4183284806" Jul 08 19:04:00.275860 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:00.275822 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26v6h" Jul 08 19:04:00.276418 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:00.275944 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26v6h" podUID="6fcd111d-1192-4c3e-90e7-2294cdec33ee" Jul 08 19:04:00.425407 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:00.425373 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jwprt" Jul 08 19:04:00.425597 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:00.425528 2558 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 08 19:04:00.426162 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:00.426142 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jwprt" Jul 08 19:04:01.276504 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:01.276259 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlw2c" Jul 08 19:04:01.277037 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:01.276272 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5rmts" Jul 08 19:04:01.277037 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:01.276637 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlw2c" podUID="640044e9-7c83-4f5a-9c9d-2e4183284806" Jul 08 19:04:01.277037 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:01.276709 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5rmts" podUID="99034b7e-72e2-4a28-8711-233630b7e9ea" Jul 08 19:04:02.276163 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:02.276126 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26v6h" Jul 08 19:04:02.276327 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:02.276237 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26v6h" podUID="6fcd111d-1192-4c3e-90e7-2294cdec33ee" Jul 08 19:04:02.645640 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:02.645619 2558 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 08 19:04:03.259082 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:03.258797 2558 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-08T19:04:02.645636865Z","UUID":"15f57c67-46bd-4f10-a7e1-f4a0ce5c1e5a","Handler":null,"Name":"","Endpoint":""} Jul 08 19:04:03.260269 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:03.260249 2558 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 08 19:04:03.260405 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:03.260282 2558 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 08 19:04:03.275907 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:03.275881 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5rmts" Jul 08 19:04:03.276024 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:03.275990 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5rmts" podUID="99034b7e-72e2-4a28-8711-233630b7e9ea" Jul 08 19:04:03.276024 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:03.276018 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlw2c" Jul 08 19:04:03.276133 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:03.276065 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlw2c" podUID="640044e9-7c83-4f5a-9c9d-2e4183284806" Jul 08 19:04:03.400644 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:03.400542 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rb7nx" event={"ID":"a1318ccd-51f3-4a63-9a79-28ebe1c65913","Type":"ContainerStarted","Data":"5d898fe9e20d5c8051e96fdb2b387b5db593d404ee24fadf52bd9bba365e5cdd"} Jul 08 19:04:03.404168 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:03.404137 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" event={"ID":"7873f48e-334b-4a54-8908-191e668b953d","Type":"ContainerStarted","Data":"71d72c4c770cc2ec00972d34fcbe7cec827039e027bd47d87c17d4699471202f"} Jul 08 19:04:03.404358 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:03.404225 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" event={"ID":"7873f48e-334b-4a54-8908-191e668b953d","Type":"ContainerStarted","Data":"b14891fa60cf6e1d33f5dbd26884713404c8074864d12fb3f08b8824976bee54"} Jul 08 19:04:03.404358 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:03.404259 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" event={"ID":"7873f48e-334b-4a54-8908-191e668b953d","Type":"ContainerStarted","Data":"a9f9e3daddcd66e2dc2c2dda7ba6e1d50190555820685881cf2ac8ca32e3ab70"} Jul 08 19:04:03.404358 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:03.404298 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" event={"ID":"7873f48e-334b-4a54-8908-191e668b953d","Type":"ContainerStarted","Data":"065ecec28d9448142ecd1746d5ace720d584fda57c3941a10f53c6ef9665064c"} Jul 08 19:04:03.404358 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:03.404343 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" event={"ID":"7873f48e-334b-4a54-8908-191e668b953d","Type":"ContainerStarted","Data":"ee9a02c0432c28b357c9cb02f65b42d36c0bff93b1313cda45fac7f1d72dd560"} Jul 08 19:04:03.404561 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:03.404388 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" event={"ID":"7873f48e-334b-4a54-8908-191e668b953d","Type":"ContainerStarted","Data":"248e5553bf6984e98596047857098fe7c66712158443f389c3f5cd2589212d32"} Jul 08 19:04:03.406349 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:03.406321 2558 generic.go:356] "Generic (PLEG): container finished" podID="60145409-2aa4-4d68-9405-30b21fae674e" containerID="40779b30a46cd69f560da3ee3a5a88bca06cd6ab50a8d04dc94f0c4c261b17bb" exitCode=0 Jul 08 19:04:03.406465 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:03.406398 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7ks2q" event={"ID":"60145409-2aa4-4d68-9405-30b21fae674e","Type":"ContainerDied","Data":"40779b30a46cd69f560da3ee3a5a88bca06cd6ab50a8d04dc94f0c4c261b17bb"} Jul 08 19:04:03.408645 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:03.408622 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glkc" event={"ID":"903e7441-02b7-49ee-a946-75875b1e34e3","Type":"ContainerStarted","Data":"e049cf1aa87471a394835629d7b8b311cd192819c8892c167030d66f6d289639"} Jul 08 19:04:03.422481 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:03.422426 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-rb7nx" podStartSLOduration=4.688169871 podStartE2EDuration="26.422411283s" totalImagesPullingTime="21.734241412s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:03:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:04:03.421906206 +0000 UTC m=+26.680859128" watchObservedRunningTime="2026-07-08 19:04:03.422411283 +0000 UTC m=+26.681364212" Jul 08 19:04:04.276024 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:04.275994 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26v6h" Jul 08 19:04:04.276629 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:04.276083 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26v6h" podUID="6fcd111d-1192-4c3e-90e7-2294cdec33ee" Jul 08 19:04:04.412430 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:04.412344 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glkc" event={"ID":"903e7441-02b7-49ee-a946-75875b1e34e3","Type":"ContainerStarted","Data":"76a1c9059cfb33ec2d124c22f7aaa4263fe9b268049fcfd6e932042ce4517ee0"} Jul 08 19:04:04.436487 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:04.436438 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9glkc" podStartSLOduration=3.368139423 podStartE2EDuration="27.436423367s" totalImagesPullingTime="24.068283944s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:03:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:04:04.436056681 +0000 UTC m=+27.695009606" watchObservedRunningTime="2026-07-08 19:04:04.436423367 +0000 UTC m=+27.695376291" Jul 08 19:04:05.275700 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:05.275679 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5rmts" Jul 08 19:04:05.275825 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:05.275679 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlw2c" Jul 08 19:04:05.275825 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:05.275782 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5rmts" podUID="99034b7e-72e2-4a28-8711-233630b7e9ea" Jul 08 19:04:05.275930 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:05.275891 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlw2c" podUID="640044e9-7c83-4f5a-9c9d-2e4183284806" Jul 08 19:04:05.418181 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:05.418138 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" event={"ID":"7873f48e-334b-4a54-8908-191e668b953d","Type":"ContainerStarted","Data":"8d85768eb025b7ab1e0594f89508d2eac2b085491cd3bad2aa894588dd58c676"} Jul 08 19:04:05.420141 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:05.420114 2558 generic.go:356] "Generic (PLEG): container finished" podID="60145409-2aa4-4d68-9405-30b21fae674e" containerID="e1702486b59f3a0e377728e4cab82c2a62868202c2172ec36d6493aaee88e6ed" exitCode=0 Jul 08 19:04:05.420243 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:05.420191 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7ks2q" event={"ID":"60145409-2aa4-4d68-9405-30b21fae674e","Type":"ContainerDied","Data":"e1702486b59f3a0e377728e4cab82c2a62868202c2172ec36d6493aaee88e6ed"} Jul 08 19:04:06.276476 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:06.276440 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26v6h" Jul 08 19:04:06.276664 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:06.276552 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26v6h" podUID="6fcd111d-1192-4c3e-90e7-2294cdec33ee" Jul 08 19:04:07.276188 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:07.276155 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5rmts" Jul 08 19:04:07.276703 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:07.276248 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5rmts" podUID="99034b7e-72e2-4a28-8711-233630b7e9ea" Jul 08 19:04:07.276703 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:07.276320 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlw2c" Jul 08 19:04:07.276703 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:07.276401 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlw2c" podUID="640044e9-7c83-4f5a-9c9d-2e4183284806" Jul 08 19:04:07.426664 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:07.426632 2558 generic.go:356] "Generic (PLEG): container finished" podID="60145409-2aa4-4d68-9405-30b21fae674e" containerID="f5cb58eebc10ae77c0eb3dceec2922d942137059fd780090183af164f9e0fec1" exitCode=0 Jul 08 19:04:07.426852 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:07.426690 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7ks2q" event={"ID":"60145409-2aa4-4d68-9405-30b21fae674e","Type":"ContainerDied","Data":"f5cb58eebc10ae77c0eb3dceec2922d942137059fd780090183af164f9e0fec1"} Jul 08 19:04:08.276433 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:08.276188 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26v6h" Jul 08 19:04:08.276832 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:08.276514 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26v6h" podUID="6fcd111d-1192-4c3e-90e7-2294cdec33ee" Jul 08 19:04:08.432129 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:08.432097 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" event={"ID":"7873f48e-334b-4a54-8908-191e668b953d","Type":"ContainerStarted","Data":"e8fa74e7b37d8bea66829e4b7c7ede267209cf352bd009c82e49aa50681416d7"} Jul 08 19:04:08.432417 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:08.432384 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:04:08.432527 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:08.432474 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:04:08.447555 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:08.447533 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:04:08.464702 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:08.464663 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" podStartSLOduration=9.697766878 podStartE2EDuration="31.464650259s" totalImagesPullingTime="21.766883381s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:03:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:04:08.462931681 +0000 UTC m=+31.721884606" watchObservedRunningTime="2026-07-08 19:04:08.464650259 +0000 UTC m=+31.723603185" Jul 08 19:04:09.276062 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:09.276027 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlw2c" Jul 08 19:04:09.276246 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:09.276025 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5rmts" Jul 08 19:04:09.276246 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:09.276170 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlw2c" podUID="640044e9-7c83-4f5a-9c9d-2e4183284806" Jul 08 19:04:09.276346 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:09.276244 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5rmts" podUID="99034b7e-72e2-4a28-8711-233630b7e9ea" Jul 08 19:04:09.434420 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:09.434383 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:04:09.448337 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:09.448316 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:04:09.938227 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:09.938150 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-26v6h"] Jul 08 19:04:09.938389 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:09.938300 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26v6h" Jul 08 19:04:09.938475 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:09.938446 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26v6h" podUID="6fcd111d-1192-4c3e-90e7-2294cdec33ee" Jul 08 19:04:09.945691 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:09.945669 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mlw2c"] Jul 08 19:04:09.945818 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:09.945778 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlw2c" Jul 08 19:04:09.945894 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:09.945873 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlw2c" podUID="640044e9-7c83-4f5a-9c9d-2e4183284806" Jul 08 19:04:09.955949 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:09.955913 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5rmts"] Jul 08 19:04:09.956093 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:09.956050 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5rmts" Jul 08 19:04:09.956183 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:09.956163 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5rmts" podUID="99034b7e-72e2-4a28-8711-233630b7e9ea" Jul 08 19:04:11.118122 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:11.118081 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/640044e9-7c83-4f5a-9c9d-2e4183284806-original-pull-secret\") pod \"global-pull-secret-syncer-mlw2c\" (UID: \"640044e9-7c83-4f5a-9c9d-2e4183284806\") " pod="kube-system/global-pull-secret-syncer-mlw2c" Jul 08 19:04:11.118509 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:11.118154 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99034b7e-72e2-4a28-8711-233630b7e9ea-metrics-certs\") pod \"network-metrics-daemon-5rmts\" (UID: \"99034b7e-72e2-4a28-8711-233630b7e9ea\") " pod="openshift-multus/network-metrics-daemon-5rmts" Jul 08 19:04:11.118509 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:11.118231 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 19:04:11.118509 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:11.118266 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:04:11.118509 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:11.118294 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/640044e9-7c83-4f5a-9c9d-2e4183284806-original-pull-secret podName:640044e9-7c83-4f5a-9c9d-2e4183284806 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:43.118280087 +0000 UTC m=+66.377233013 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/640044e9-7c83-4f5a-9c9d-2e4183284806-original-pull-secret") pod "global-pull-secret-syncer-mlw2c" (UID: "640044e9-7c83-4f5a-9c9d-2e4183284806") : object "kube-system"/"original-pull-secret" not registered Jul 08 19:04:11.118509 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:11.118317 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99034b7e-72e2-4a28-8711-233630b7e9ea-metrics-certs podName:99034b7e-72e2-4a28-8711-233630b7e9ea nodeName:}" failed. No retries permitted until 2026-07-08 19:04:43.118301615 +0000 UTC m=+66.377254518 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99034b7e-72e2-4a28-8711-233630b7e9ea-metrics-certs") pod "network-metrics-daemon-5rmts" (UID: "99034b7e-72e2-4a28-8711-233630b7e9ea") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:04:11.219467 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:11.219431 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lszs7\" (UniqueName: \"kubernetes.io/projected/6fcd111d-1192-4c3e-90e7-2294cdec33ee-kube-api-access-lszs7\") pod \"network-check-target-26v6h\" (UID: \"6fcd111d-1192-4c3e-90e7-2294cdec33ee\") " pod="openshift-network-diagnostics/network-check-target-26v6h" Jul 08 19:04:11.219679 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:11.219661 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 19:04:11.219745 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:11.219687 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 19:04:11.219745 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:11.219699 2558 projected.go:196] Error preparing data for projected volume kube-api-access-lszs7 for pod openshift-network-diagnostics/network-check-target-26v6h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:04:11.219822 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:11.219759 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fcd111d-1192-4c3e-90e7-2294cdec33ee-kube-api-access-lszs7 podName:6fcd111d-1192-4c3e-90e7-2294cdec33ee nodeName:}" failed. No retries permitted until 2026-07-08 19:04:43.219744608 +0000 UTC m=+66.478697512 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lszs7" (UniqueName: "kubernetes.io/projected/6fcd111d-1192-4c3e-90e7-2294cdec33ee-kube-api-access-lszs7") pod "network-check-target-26v6h" (UID: "6fcd111d-1192-4c3e-90e7-2294cdec33ee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:04:11.275598 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:11.275562 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26v6h" Jul 08 19:04:11.275728 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:11.275563 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlw2c" Jul 08 19:04:11.275728 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:11.275665 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26v6h" podUID="6fcd111d-1192-4c3e-90e7-2294cdec33ee" Jul 08 19:04:11.275728 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:11.275690 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5rmts" Jul 08 19:04:11.275843 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:11.275787 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlw2c" podUID="640044e9-7c83-4f5a-9c9d-2e4183284806" Jul 08 19:04:11.275922 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:11.275903 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5rmts" podUID="99034b7e-72e2-4a28-8711-233630b7e9ea" Jul 08 19:04:13.276165 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:13.276090 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5rmts" Jul 08 19:04:13.276165 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:13.276116 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26v6h" Jul 08 19:04:13.276635 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:13.276092 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlw2c" Jul 08 19:04:13.276635 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:13.276204 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5rmts" podUID="99034b7e-72e2-4a28-8711-233630b7e9ea" Jul 08 19:04:13.276635 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:13.276284 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26v6h" podUID="6fcd111d-1192-4c3e-90e7-2294cdec33ee" Jul 08 19:04:13.276635 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:13.276353 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlw2c" podUID="640044e9-7c83-4f5a-9c9d-2e4183284806" Jul 08 19:04:14.448351 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:14.448268 2558 generic.go:356] "Generic (PLEG): container finished" podID="60145409-2aa4-4d68-9405-30b21fae674e" containerID="c8388fbf6600f2a73c233e0edaa3f99478e4b57481455b909de1cbd6eadf74d4" exitCode=0 Jul 08 19:04:14.448351 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:14.448333 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7ks2q" event={"ID":"60145409-2aa4-4d68-9405-30b21fae674e","Type":"ContainerDied","Data":"c8388fbf6600f2a73c233e0edaa3f99478e4b57481455b909de1cbd6eadf74d4"} Jul 08 19:04:15.012032 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.012005 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-191.ec2.internal" event="NodeReady" Jul 08 19:04:15.012147 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.012141 2558 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 08 19:04:15.068668 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.068154 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7c4688bdd8-xlx9p"] Jul 08 19:04:15.083072 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.083053 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c4688bdd8-xlx9p" Jul 08 19:04:15.089867 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.089843 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 08 19:04:15.089971 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.089879 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zm6q7\"" Jul 08 19:04:15.090392 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.090379 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 08 19:04:15.093782 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.093765 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 08 19:04:15.103042 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.103026 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 08 19:04:15.114296 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.114258 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c4688bdd8-xlx9p"] Jul 08 19:04:15.129906 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.129887 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lng6h"] Jul 08 19:04:15.142634 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.142613 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ndtk5"] Jul 08 19:04:15.142731 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.142715 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lng6h" Jul 08 19:04:15.146254 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.146192 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 08 19:04:15.146254 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.146203 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5l8cc\"" Jul 08 19:04:15.146375 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.146336 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 08 19:04:15.147216 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.147192 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-image-registry-private-configuration\") pod \"image-registry-7c4688bdd8-xlx9p\" (UID: \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\") " pod="openshift-image-registry/image-registry-7c4688bdd8-xlx9p" Jul 08 19:04:15.147325 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.147274 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-installation-pull-secrets\") pod \"image-registry-7c4688bdd8-xlx9p\" (UID: \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\") " pod="openshift-image-registry/image-registry-7c4688bdd8-xlx9p" Jul 08 19:04:15.147325 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.147294 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 08 19:04:15.147325 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.147319 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-bound-sa-token\") pod \"image-registry-7c4688bdd8-xlx9p\" (UID: \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\") " pod="openshift-image-registry/image-registry-7c4688bdd8-xlx9p" Jul 08 19:04:15.147503 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.147419 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-registry-tls\") pod \"image-registry-7c4688bdd8-xlx9p\" (UID: \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\") " pod="openshift-image-registry/image-registry-7c4688bdd8-xlx9p" Jul 08 19:04:15.147503 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.147459 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-ca-trust-extracted\") pod \"image-registry-7c4688bdd8-xlx9p\" (UID: \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\") " pod="openshift-image-registry/image-registry-7c4688bdd8-xlx9p" Jul 08 19:04:15.147503 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.147480 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-registry-certificates\") pod \"image-registry-7c4688bdd8-xlx9p\" (UID: \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\") " pod="openshift-image-registry/image-registry-7c4688bdd8-xlx9p" Jul 08 19:04:15.147676 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.147539 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-trusted-ca\") pod \"image-registry-7c4688bdd8-xlx9p\" (UID: \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\") " pod="openshift-image-registry/image-registry-7c4688bdd8-xlx9p" Jul 08 19:04:15.147676 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.147623 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwcbw\" (UniqueName: \"kubernetes.io/projected/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-kube-api-access-jwcbw\") pod \"image-registry-7c4688bdd8-xlx9p\" (UID: \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\") " pod="openshift-image-registry/image-registry-7c4688bdd8-xlx9p" Jul 08 19:04:15.155865 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.155841 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lng6h"] Jul 08 19:04:15.155978 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.155961 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ndtk5" Jul 08 19:04:15.161478 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.161460 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xm8m6\"" Jul 08 19:04:15.162384 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.162371 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 08 19:04:15.162459 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.162440 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 08 19:04:15.163981 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.163962 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ndtk5"] Jul 08 19:04:15.248025 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.247955 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eae871c8-0143-47c6-853c-2413d8f6fa70-tmp-dir\") pod \"dns-default-ndtk5\" (UID: \"eae871c8-0143-47c6-853c-2413d8f6fa70\") " pod="openshift-dns/dns-default-ndtk5" Jul 08 19:04:15.248025 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.248004 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knjvn\" (UniqueName: \"kubernetes.io/projected/2e731fad-40cc-4fa8-92cf-5465ff9a29dd-kube-api-access-knjvn\") pod \"ingress-canary-lng6h\" (UID: \"2e731fad-40cc-4fa8-92cf-5465ff9a29dd\") " pod="openshift-ingress-canary/ingress-canary-lng6h" Jul 08 19:04:15.248025 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.248021 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eae871c8-0143-47c6-853c-2413d8f6fa70-metrics-tls\") pod \"dns-default-ndtk5\" (UID: \"eae871c8-0143-47c6-853c-2413d8f6fa70\") " pod="openshift-dns/dns-default-ndtk5" Jul 08 19:04:15.248282 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.248042 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-image-registry-private-configuration\") pod \"image-registry-7c4688bdd8-xlx9p\" (UID: \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\") " pod="openshift-image-registry/image-registry-7c4688bdd8-xlx9p" Jul 08 19:04:15.248282 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.248060 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-installation-pull-secrets\") pod \"image-registry-7c4688bdd8-xlx9p\" (UID: \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\") " pod="openshift-image-registry/image-registry-7c4688bdd8-xlx9p" Jul 08 19:04:15.248282 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.248104 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-bound-sa-token\") pod \"image-registry-7c4688bdd8-xlx9p\" (UID: \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\") " pod="openshift-image-registry/image-registry-7c4688bdd8-xlx9p" Jul 08 19:04:15.248282 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.248147 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-registry-tls\") pod \"image-registry-7c4688bdd8-xlx9p\" (UID: \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\") " pod="openshift-image-registry/image-registry-7c4688bdd8-xlx9p" Jul 08 19:04:15.248282 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.248175 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-ca-trust-extracted\") pod \"image-registry-7c4688bdd8-xlx9p\" (UID: \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\") " pod="openshift-image-registry/image-registry-7c4688bdd8-xlx9p" Jul 08 19:04:15.248282 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.248199 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-registry-certificates\") pod \"image-registry-7c4688bdd8-xlx9p\" (UID: \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\") " pod="openshift-image-registry/image-registry-7c4688bdd8-xlx9p" Jul 08 19:04:15.248282 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:15.248223 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 19:04:15.248282 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:15.248244 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c4688bdd8-xlx9p: secret "image-registry-tls" not found Jul 08 19:04:15.248666 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:15.248312 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-registry-tls podName:fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:15.748291171 +0000 UTC m=+39.007244077 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-registry-tls") pod "image-registry-7c4688bdd8-xlx9p" (UID: "fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7") : secret "image-registry-tls" not found Jul 08 19:04:15.248666 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.248226 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e731fad-40cc-4fa8-92cf-5465ff9a29dd-cert\") pod \"ingress-canary-lng6h\" (UID: \"2e731fad-40cc-4fa8-92cf-5465ff9a29dd\") " pod="openshift-ingress-canary/ingress-canary-lng6h" Jul 08 19:04:15.248666 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.248447 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/eae871c8-0143-47c6-853c-2413d8f6fa70-config-volume\") pod \"dns-default-ndtk5\" (UID: \"eae871c8-0143-47c6-853c-2413d8f6fa70\") " pod="openshift-dns/dns-default-ndtk5" Jul 08 19:04:15.248666 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.248518 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kgcb\" (UniqueName: \"kubernetes.io/projected/eae871c8-0143-47c6-853c-2413d8f6fa70-kube-api-access-4kgcb\") pod \"dns-default-ndtk5\" (UID: \"eae871c8-0143-47c6-853c-2413d8f6fa70\") " pod="openshift-dns/dns-default-ndtk5" Jul 08 19:04:15.248666 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.248550 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-trusted-ca\") pod \"image-registry-7c4688bdd8-xlx9p\" (UID: \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\") " pod="openshift-image-registry/image-registry-7c4688bdd8-xlx9p" Jul 08 19:04:15.248666 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.248636 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-ca-trust-extracted\") pod \"image-registry-7c4688bdd8-xlx9p\" (UID: \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\") " pod="openshift-image-registry/image-registry-7c4688bdd8-xlx9p" Jul 08 19:04:15.248930 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.248762 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwcbw\" (UniqueName: \"kubernetes.io/projected/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-kube-api-access-jwcbw\") pod \"image-registry-7c4688bdd8-xlx9p\" (UID: \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\") " pod="openshift-image-registry/image-registry-7c4688bdd8-xlx9p" Jul 08 19:04:15.248930 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.248833 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-registry-certificates\") pod \"image-registry-7c4688bdd8-xlx9p\" (UID: \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\") " pod="openshift-image-registry/image-registry-7c4688bdd8-xlx9p" Jul 08 19:04:15.249538 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.249514 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-trusted-ca\") pod \"image-registry-7c4688bdd8-xlx9p\" (UID: \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\") " pod="openshift-image-registry/image-registry-7c4688bdd8-xlx9p" Jul 08 19:04:15.252009 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.251989 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-image-registry-private-configuration\") pod \"image-registry-7c4688bdd8-xlx9p\" (UID: \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\") " pod="openshift-image-registry/image-registry-7c4688bdd8-xlx9p" Jul 08 19:04:15.252009 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.251997 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-installation-pull-secrets\") pod \"image-registry-7c4688bdd8-xlx9p\" (UID: \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\") " pod="openshift-image-registry/image-registry-7c4688bdd8-xlx9p" Jul 08 19:04:15.269672 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.269644 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-bound-sa-token\") pod \"image-registry-7c4688bdd8-xlx9p\" (UID: \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\") " pod="openshift-image-registry/image-registry-7c4688bdd8-xlx9p" Jul 08 19:04:15.271436 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.271415 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwcbw\" (UniqueName: \"kubernetes.io/projected/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-kube-api-access-jwcbw\") pod \"image-registry-7c4688bdd8-xlx9p\" (UID: \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\") " pod="openshift-image-registry/image-registry-7c4688bdd8-xlx9p" Jul 08 19:04:15.276365 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.276347 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26v6h" Jul 08 19:04:15.276456 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.276364 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5rmts" Jul 08 19:04:15.276649 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.276627 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlw2c" Jul 08 19:04:15.283223 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.283204 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 08 19:04:15.283300 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.283227 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gzqsf\"" Jul 08 19:04:15.283382 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.283367 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 08 19:04:15.283755 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.283738 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 08 19:04:15.284176 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.284163 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 08 19:04:15.284248 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.284234 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kpjrj\"" Jul 08 19:04:15.349692 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.349651 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eae871c8-0143-47c6-853c-2413d8f6fa70-tmp-dir\") pod \"dns-default-ndtk5\" (UID: \"eae871c8-0143-47c6-853c-2413d8f6fa70\") " pod="openshift-dns/dns-default-ndtk5" Jul 08 19:04:15.349829 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.349706 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knjvn\" (UniqueName: \"kubernetes.io/projected/2e731fad-40cc-4fa8-92cf-5465ff9a29dd-kube-api-access-knjvn\") pod \"ingress-canary-lng6h\" (UID: \"2e731fad-40cc-4fa8-92cf-5465ff9a29dd\") " pod="openshift-ingress-canary/ingress-canary-lng6h" Jul 08 19:04:15.349866 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.349821 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eae871c8-0143-47c6-853c-2413d8f6fa70-metrics-tls\") pod \"dns-default-ndtk5\" (UID: \"eae871c8-0143-47c6-853c-2413d8f6fa70\") " pod="openshift-dns/dns-default-ndtk5" Jul 08 19:04:15.349900 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.349891 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e731fad-40cc-4fa8-92cf-5465ff9a29dd-cert\") pod \"ingress-canary-lng6h\" (UID: \"2e731fad-40cc-4fa8-92cf-5465ff9a29dd\") " pod="openshift-ingress-canary/ingress-canary-lng6h" Jul 08 19:04:15.349929 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.349917 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/eae871c8-0143-47c6-853c-2413d8f6fa70-config-volume\") pod \"dns-default-ndtk5\" (UID: \"eae871c8-0143-47c6-853c-2413d8f6fa70\") " pod="openshift-dns/dns-default-ndtk5" Jul 08 19:04:15.349982 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.349959 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kgcb\" (UniqueName: \"kubernetes.io/projected/eae871c8-0143-47c6-853c-2413d8f6fa70-kube-api-access-4kgcb\") pod \"dns-default-ndtk5\" (UID: \"eae871c8-0143-47c6-853c-2413d8f6fa70\") " pod="openshift-dns/dns-default-ndtk5" Jul 08 19:04:15.350028 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:15.349983 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 19:04:15.350028 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.350011 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eae871c8-0143-47c6-853c-2413d8f6fa70-tmp-dir\") pod \"dns-default-ndtk5\" (UID: \"eae871c8-0143-47c6-853c-2413d8f6fa70\") " pod="openshift-dns/dns-default-ndtk5" Jul 08 19:04:15.350117 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:15.350045 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eae871c8-0143-47c6-853c-2413d8f6fa70-metrics-tls podName:eae871c8-0143-47c6-853c-2413d8f6fa70 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:15.850027125 +0000 UTC m=+39.108980030 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eae871c8-0143-47c6-853c-2413d8f6fa70-metrics-tls") pod "dns-default-ndtk5" (UID: "eae871c8-0143-47c6-853c-2413d8f6fa70") : secret "dns-default-metrics-tls" not found Jul 08 19:04:15.350428 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.350409 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/eae871c8-0143-47c6-853c-2413d8f6fa70-config-volume\") pod \"dns-default-ndtk5\" (UID: \"eae871c8-0143-47c6-853c-2413d8f6fa70\") " pod="openshift-dns/dns-default-ndtk5" Jul 08 19:04:15.352225 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.352204 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e731fad-40cc-4fa8-92cf-5465ff9a29dd-cert\") pod \"ingress-canary-lng6h\" (UID: \"2e731fad-40cc-4fa8-92cf-5465ff9a29dd\") " pod="openshift-ingress-canary/ingress-canary-lng6h" Jul 08 19:04:15.365132 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.365106 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knjvn\" (UniqueName: \"kubernetes.io/projected/2e731fad-40cc-4fa8-92cf-5465ff9a29dd-kube-api-access-knjvn\") pod \"ingress-canary-lng6h\" (UID: \"2e731fad-40cc-4fa8-92cf-5465ff9a29dd\") " pod="openshift-ingress-canary/ingress-canary-lng6h" Jul 08 19:04:15.369923 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.369898 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kgcb\" (UniqueName: \"kubernetes.io/projected/eae871c8-0143-47c6-853c-2413d8f6fa70-kube-api-access-4kgcb\") pod \"dns-default-ndtk5\" (UID: \"eae871c8-0143-47c6-853c-2413d8f6fa70\") " pod="openshift-dns/dns-default-ndtk5" Jul 08 19:04:15.408128 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.408094 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:15.453206 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.453185 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lng6h" Jul 08 19:04:15.453855 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.453750 2558 generic.go:356] "Generic (PLEG): container finished" podID="60145409-2aa4-4d68-9405-30b21fae674e" containerID="b73c08ba04c4625a3ee5fea24f8ced5eaa5ba5066c99ac12cd4379ef562d5f2a" exitCode=0 Jul 08 19:04:15.453855 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.453812 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7ks2q" event={"ID":"60145409-2aa4-4d68-9405-30b21fae674e","Type":"ContainerDied","Data":"b73c08ba04c4625a3ee5fea24f8ced5eaa5ba5066c99ac12cd4379ef562d5f2a"} Jul 08 19:04:15.550848 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.550592 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:15.619686 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.619659 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lng6h"] Jul 08 19:04:15.624298 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:04:15.624271 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e731fad_40cc_4fa8_92cf_5465ff9a29dd.slice/crio-345b9e226c2f4c1d0cb8d8c384da3f29d6ea44d131734b70f6fd559af5fe4007 WatchSource:0}: Error finding container 345b9e226c2f4c1d0cb8d8c384da3f29d6ea44d131734b70f6fd559af5fe4007: Status 404 returned error can't find the container with id 345b9e226c2f4c1d0cb8d8c384da3f29d6ea44d131734b70f6fd559af5fe4007 Jul 08 19:04:15.662113 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.662082 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:15.753363 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.753328 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-registry-tls\") pod \"image-registry-7c4688bdd8-xlx9p\" (UID: \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\") " pod="openshift-image-registry/image-registry-7c4688bdd8-xlx9p" Jul 08 19:04:15.753521 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:15.753479 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 19:04:15.753521 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:15.753497 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c4688bdd8-xlx9p: secret "image-registry-tls" not found Jul 08 19:04:15.753611 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:15.753553 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-registry-tls podName:fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:16.753537901 +0000 UTC m=+40.012490807 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-registry-tls") pod "image-registry-7c4688bdd8-xlx9p" (UID: "fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7") : secret "image-registry-tls" not found Jul 08 19:04:15.854454 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.854361 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eae871c8-0143-47c6-853c-2413d8f6fa70-metrics-tls\") pod \"dns-default-ndtk5\" (UID: \"eae871c8-0143-47c6-853c-2413d8f6fa70\") " pod="openshift-dns/dns-default-ndtk5" Jul 08 19:04:15.854622 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:15.854504 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 19:04:15.854622 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:15.854561 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eae871c8-0143-47c6-853c-2413d8f6fa70-metrics-tls podName:eae871c8-0143-47c6-853c-2413d8f6fa70 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:16.8545468 +0000 UTC m=+40.113499702 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eae871c8-0143-47c6-853c-2413d8f6fa70-metrics-tls") pod "dns-default-ndtk5" (UID: "eae871c8-0143-47c6-853c-2413d8f6fa70") : secret "dns-default-metrics-tls" not found Jul 08 19:04:15.908723 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.908689 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:15.984966 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:15.984942 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:16.390039 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:16.390006 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:16.459737 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:16.459702 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7ks2q" event={"ID":"60145409-2aa4-4d68-9405-30b21fae674e","Type":"ContainerStarted","Data":"ec4b7d03b0a7ee4fc11e8d8c4ea6d112958e6cfeedc0b077f273f3818208234b"} Jul 08 19:04:16.460807 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:16.460786 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lng6h" event={"ID":"2e731fad-40cc-4fa8-92cf-5465ff9a29dd","Type":"ContainerStarted","Data":"345b9e226c2f4c1d0cb8d8c384da3f29d6ea44d131734b70f6fd559af5fe4007"} Jul 08 19:04:16.489955 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:16.489921 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:16.493939 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:16.493884 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7ks2q" podStartSLOduration=7.875906959 podStartE2EDuration="39.493866208s" totalImagesPullingTime="31.617959249s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:03:37 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:04:16.493767413 +0000 UTC m=+39.752720339" watchObservedRunningTime="2026-07-08 19:04:16.493866208 +0000 UTC m=+39.752819126" Jul 08 19:04:16.760206 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:16.760122 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-registry-tls\") pod \"image-registry-7c4688bdd8-xlx9p\" (UID: \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\") " pod="openshift-image-registry/image-registry-7c4688bdd8-xlx9p" Jul 08 19:04:16.760345 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:16.760266 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 19:04:16.760345 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:16.760283 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c4688bdd8-xlx9p: secret "image-registry-tls" not found Jul 08 19:04:16.760345 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:16.760336 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-registry-tls podName:fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:18.760322071 +0000 UTC m=+42.019274979 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-registry-tls") pod "image-registry-7c4688bdd8-xlx9p" (UID: "fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7") : secret "image-registry-tls" not found Jul 08 19:04:16.785959 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:16.785918 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:16.861212 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:16.861179 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eae871c8-0143-47c6-853c-2413d8f6fa70-metrics-tls\") pod \"dns-default-ndtk5\" (UID: \"eae871c8-0143-47c6-853c-2413d8f6fa70\") " pod="openshift-dns/dns-default-ndtk5" Jul 08 19:04:16.861347 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:16.861332 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 19:04:16.861414 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:16.861403 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eae871c8-0143-47c6-853c-2413d8f6fa70-metrics-tls podName:eae871c8-0143-47c6-853c-2413d8f6fa70 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:18.861388068 +0000 UTC m=+42.120340972 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eae871c8-0143-47c6-853c-2413d8f6fa70-metrics-tls") pod "dns-default-ndtk5" (UID: "eae871c8-0143-47c6-853c-2413d8f6fa70") : secret "dns-default-metrics-tls" not found Jul 08 19:04:16.887336 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:16.887303 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:17.592767 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:17.592719 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:17.694262 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:17.694226 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:17.999858 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:17.999822 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:18.148545 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:18.148516 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:18.776836 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:18.776802 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-registry-tls\") pod \"image-registry-7c4688bdd8-xlx9p\" (UID: \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\") " pod="openshift-image-registry/image-registry-7c4688bdd8-xlx9p" Jul 08 19:04:18.777265 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:18.776921 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 19:04:18.777265 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:18.776939 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c4688bdd8-xlx9p: secret "image-registry-tls" not found Jul 08 19:04:18.777265 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:18.776998 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-registry-tls podName:fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:22.77698196 +0000 UTC m=+46.035934865 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-registry-tls") pod "image-registry-7c4688bdd8-xlx9p" (UID: "fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7") : secret "image-registry-tls" not found Jul 08 19:04:18.829976 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:18.829942 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:18.877967 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:18.877929 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eae871c8-0143-47c6-853c-2413d8f6fa70-metrics-tls\") pod \"dns-default-ndtk5\" (UID: \"eae871c8-0143-47c6-853c-2413d8f6fa70\") " pod="openshift-dns/dns-default-ndtk5" Jul 08 19:04:18.878121 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:18.878102 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 19:04:18.878180 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:18.878174 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eae871c8-0143-47c6-853c-2413d8f6fa70-metrics-tls podName:eae871c8-0143-47c6-853c-2413d8f6fa70 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:22.878154576 +0000 UTC m=+46.137107486 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eae871c8-0143-47c6-853c-2413d8f6fa70-metrics-tls") pod "dns-default-ndtk5" (UID: "eae871c8-0143-47c6-853c-2413d8f6fa70") : secret "dns-default-metrics-tls" not found Jul 08 19:04:18.930207 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:18.930178 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:19.185424 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:19.185392 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:19.284883 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:19.284851 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:19.469559 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:19.469411 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lng6h" event={"ID":"2e731fad-40cc-4fa8-92cf-5465ff9a29dd","Type":"ContainerStarted","Data":"c31ec23ef45557feed4a2db3a425da77777a01d7882e52d7acaa1d2e1209198e"} Jul 08 19:04:19.487436 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:19.487385 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lng6h" podStartSLOduration=0.807082347 podStartE2EDuration="4.487369428s" totalImagesPullingTime="3.680287081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:04:19.487246277 +0000 UTC m=+42.746199192" watchObservedRunningTime="2026-07-08 19:04:19.487369428 +0000 UTC m=+42.746322351" Jul 08 19:04:21.906674 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:21.906636 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-54hnr"] Jul 08 19:04:21.929690 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:21.929668 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-54hnr"] Jul 08 19:04:21.929815 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:21.929757 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-54hnr" Jul 08 19:04:21.935301 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:21.935283 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 08 19:04:21.936938 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:21.936920 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 08 19:04:21.937031 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:21.936969 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 08 19:04:21.937232 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:21.937215 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-s5vhg\"" Jul 08 19:04:21.937232 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:21.937226 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 08 19:04:21.941455 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:21.941437 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 08 19:04:22.000136 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.000103 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a-trusted-ca\") pod \"console-operator-68b757fb67-54hnr\" (UID: \"1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a\") " pod="openshift-console-operator/console-operator-68b757fb67-54hnr" Jul 08 19:04:22.000136 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.000141 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a-serving-cert\") pod \"console-operator-68b757fb67-54hnr\" (UID: \"1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a\") " pod="openshift-console-operator/console-operator-68b757fb67-54hnr" Jul 08 19:04:22.000327 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.000159 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9r4h\" (UniqueName: \"kubernetes.io/projected/1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a-kube-api-access-t9r4h\") pod \"console-operator-68b757fb67-54hnr\" (UID: \"1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a\") " pod="openshift-console-operator/console-operator-68b757fb67-54hnr" Jul 08 19:04:22.000327 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.000240 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a-config\") pod \"console-operator-68b757fb67-54hnr\" (UID: \"1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a\") " pod="openshift-console-operator/console-operator-68b757fb67-54hnr" Jul 08 19:04:22.011180 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.011155 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-fpx7q"] Jul 08 19:04:22.032122 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.032101 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-7qzsf"] Jul 08 19:04:22.032250 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.032238 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-fpx7q" Jul 08 19:04:22.037236 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.037215 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 08 19:04:22.037862 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.037843 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 08 19:04:22.037960 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.037862 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-tp9xs\"" Jul 08 19:04:22.038109 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.038095 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 08 19:04:22.038172 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.038124 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 08 19:04:22.050215 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.050197 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-69gjd"] Jul 08 19:04:22.050352 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.050334 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-7qzsf" Jul 08 19:04:22.054496 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.054477 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 08 19:04:22.054612 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.054482 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 08 19:04:22.056458 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.056439 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-xhgrj\"" Jul 08 19:04:22.056712 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.056697 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 08 19:04:22.056769 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.056707 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 08 19:04:22.074322 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.074301 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-fpx7q"] Jul 08 19:04:22.074400 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.074330 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-69gjd"] Jul 08 19:04:22.074400 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.074343 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-7qzsf"] Jul 08 19:04:22.074400 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.074357 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-l4fvs"] Jul 08 19:04:22.074497 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.074413 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-69gjd" Jul 08 19:04:22.077451 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.077432 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-92c5x\"" Jul 08 19:04:22.077532 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.077482 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 08 19:04:22.077532 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.077494 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 08 19:04:22.096379 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.096357 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-l4fvs" Jul 08 19:04:22.099171 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.099150 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 08 19:04:22.099171 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.099162 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 08 19:04:22.099313 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.099195 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 08 19:04:22.099313 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.099277 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 08 19:04:22.099313 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.099288 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-xrj72\"" Jul 08 19:04:22.100648 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.100631 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e5f10566-2d90-4c76-846b-8114279ad692-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-fpx7q\" (UID: \"e5f10566-2d90-4c76-846b-8114279ad692\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-fpx7q" Jul 08 19:04:22.100731 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.100661 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a-config\") pod \"console-operator-68b757fb67-54hnr\" (UID: \"1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a\") " pod="openshift-console-operator/console-operator-68b757fb67-54hnr" Jul 08 19:04:22.100731 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.100708 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/58ce9cbf-4ed4-4482-8786-d92dd2f9a0e0-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-7qzsf\" (UID: \"58ce9cbf-4ed4-4482-8786-d92dd2f9a0e0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-7qzsf" Jul 08 19:04:22.100826 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.100810 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/58ce9cbf-4ed4-4482-8786-d92dd2f9a0e0-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-7qzsf\" (UID: \"58ce9cbf-4ed4-4482-8786-d92dd2f9a0e0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-7qzsf" Jul 08 19:04:22.100878 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.100845 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2xdq\" (UniqueName: \"kubernetes.io/projected/58ce9cbf-4ed4-4482-8786-d92dd2f9a0e0-kube-api-access-q2xdq\") pod \"kube-storage-version-migrator-operator-846bf668f7-7qzsf\" (UID: \"58ce9cbf-4ed4-4482-8786-d92dd2f9a0e0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-7qzsf" Jul 08 19:04:22.100932 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.100904 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a-trusted-ca\") pod \"console-operator-68b757fb67-54hnr\" (UID: \"1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a\") " pod="openshift-console-operator/console-operator-68b757fb67-54hnr" Jul 08 19:04:22.101009 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.100938 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e5f10566-2d90-4c76-846b-8114279ad692-config\") pod \"service-ca-operator-5d46d9ffd4-fpx7q\" (UID: \"e5f10566-2d90-4c76-846b-8114279ad692\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-fpx7q" Jul 08 19:04:22.101009 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.100961 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w27z\" (UniqueName: \"kubernetes.io/projected/e5f10566-2d90-4c76-846b-8114279ad692-kube-api-access-4w27z\") pod \"service-ca-operator-5d46d9ffd4-fpx7q\" (UID: \"e5f10566-2d90-4c76-846b-8114279ad692\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-fpx7q" Jul 08 19:04:22.101009 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.100985 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwjvg\" (UniqueName: \"kubernetes.io/projected/3fd49589-6210-4804-bccd-079d96770654-kube-api-access-bwjvg\") pod \"volume-data-source-validator-5d456fdbcd-69gjd\" (UID: \"3fd49589-6210-4804-bccd-079d96770654\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-69gjd" Jul 08 19:04:22.101226 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.101015 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a-serving-cert\") pod \"console-operator-68b757fb67-54hnr\" (UID: \"1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a\") " pod="openshift-console-operator/console-operator-68b757fb67-54hnr" Jul 08 19:04:22.101226 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.101043 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9r4h\" (UniqueName: \"kubernetes.io/projected/1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a-kube-api-access-t9r4h\") pod \"console-operator-68b757fb67-54hnr\" (UID: \"1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a\") " pod="openshift-console-operator/console-operator-68b757fb67-54hnr" Jul 08 19:04:22.101463 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.101445 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a-config\") pod \"console-operator-68b757fb67-54hnr\" (UID: \"1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a\") " pod="openshift-console-operator/console-operator-68b757fb67-54hnr" Jul 08 19:04:22.102004 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.101986 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a-trusted-ca\") pod \"console-operator-68b757fb67-54hnr\" (UID: \"1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a\") " pod="openshift-console-operator/console-operator-68b757fb67-54hnr" Jul 08 19:04:22.104748 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.104685 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-l4fvs"] Jul 08 19:04:22.105930 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.105904 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a-serving-cert\") pod \"console-operator-68b757fb67-54hnr\" (UID: \"1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a\") " pod="openshift-console-operator/console-operator-68b757fb67-54hnr" Jul 08 19:04:22.106453 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.106435 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 08 19:04:22.114057 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.114029 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9r4h\" (UniqueName: \"kubernetes.io/projected/1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a-kube-api-access-t9r4h\") pod \"console-operator-68b757fb67-54hnr\" (UID: \"1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a\") " pod="openshift-console-operator/console-operator-68b757fb67-54hnr" Jul 08 19:04:22.129908 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.129883 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vvggh"] Jul 08 19:04:22.152597 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.150973 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-s5dzj"] Jul 08 19:04:22.152597 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.151685 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vvggh" Jul 08 19:04:22.157308 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.156966 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 08 19:04:22.157308 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.157161 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-nnzht\"" Jul 08 19:04:22.157457 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.157419 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 08 19:04:22.157717 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.157693 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 08 19:04:22.169184 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.169164 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7556c5577f-p4h6b"] Jul 08 19:04:22.169305 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.169288 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-s5dzj" Jul 08 19:04:22.171972 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.171949 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-8jhtf\"" Jul 08 19:04:22.193447 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.193410 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-r7czv"] Jul 08 19:04:22.193606 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.193521 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7556c5577f-p4h6b" Jul 08 19:04:22.196594 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.196557 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 08 19:04:22.196711 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.196592 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 08 19:04:22.196711 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.196602 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 08 19:04:22.196711 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.196605 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 08 19:04:22.196711 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.196659 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 08 19:04:22.196977 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.196962 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-gxtbn\"" Jul 08 19:04:22.197046 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.196997 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 08 19:04:22.201937 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.201708 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f2932e3-5510-44d8-bd92-76ebdac8ce3f-serving-cert\") pod \"insights-operator-5c847dcfc4-l4fvs\" (UID: \"9f2932e3-5510-44d8-bd92-76ebdac8ce3f\") " pod="openshift-insights/insights-operator-5c847dcfc4-l4fvs" Jul 08 19:04:22.201937 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.201779 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w49k\" (UniqueName: \"kubernetes.io/projected/812854b5-4391-45db-b96d-cddb09c0d324-kube-api-access-8w49k\") pod \"cluster-samples-operator-56fcf86bf-vvggh\" (UID: \"812854b5-4391-45db-b96d-cddb09c0d324\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vvggh" Jul 08 19:04:22.201937 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.201816 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e5f10566-2d90-4c76-846b-8114279ad692-config\") pod \"service-ca-operator-5d46d9ffd4-fpx7q\" (UID: \"e5f10566-2d90-4c76-846b-8114279ad692\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-fpx7q" Jul 08 19:04:22.201937 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.201850 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwjvg\" (UniqueName: \"kubernetes.io/projected/3fd49589-6210-4804-bccd-079d96770654-kube-api-access-bwjvg\") pod \"volume-data-source-validator-5d456fdbcd-69gjd\" (UID: \"3fd49589-6210-4804-bccd-079d96770654\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-69gjd" Jul 08 19:04:22.201937 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.201880 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e5f10566-2d90-4c76-846b-8114279ad692-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-fpx7q\" (UID: \"e5f10566-2d90-4c76-846b-8114279ad692\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-fpx7q" Jul 08 19:04:22.201937 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.201935 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/58ce9cbf-4ed4-4482-8786-d92dd2f9a0e0-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-7qzsf\" (UID: \"58ce9cbf-4ed4-4482-8786-d92dd2f9a0e0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-7qzsf" Jul 08 19:04:22.202246 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.201964 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f2932e3-5510-44d8-bd92-76ebdac8ce3f-tmp\") pod \"insights-operator-5c847dcfc4-l4fvs\" (UID: \"9f2932e3-5510-44d8-bd92-76ebdac8ce3f\") " pod="openshift-insights/insights-operator-5c847dcfc4-l4fvs" Jul 08 19:04:22.202246 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.201996 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f2932e3-5510-44d8-bd92-76ebdac8ce3f-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-l4fvs\" (UID: \"9f2932e3-5510-44d8-bd92-76ebdac8ce3f\") " pod="openshift-insights/insights-operator-5c847dcfc4-l4fvs" Jul 08 19:04:22.202246 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.202027 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhfm2\" (UniqueName: \"kubernetes.io/projected/9f2932e3-5510-44d8-bd92-76ebdac8ce3f-kube-api-access-fhfm2\") pod \"insights-operator-5c847dcfc4-l4fvs\" (UID: \"9f2932e3-5510-44d8-bd92-76ebdac8ce3f\") " pod="openshift-insights/insights-operator-5c847dcfc4-l4fvs" Jul 08 19:04:22.202246 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.202077 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2xdq\" (UniqueName: \"kubernetes.io/projected/58ce9cbf-4ed4-4482-8786-d92dd2f9a0e0-kube-api-access-q2xdq\") pod \"kube-storage-version-migrator-operator-846bf668f7-7qzsf\" (UID: \"58ce9cbf-4ed4-4482-8786-d92dd2f9a0e0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-7qzsf" Jul 08 19:04:22.202246 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.202113 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w27z\" (UniqueName: \"kubernetes.io/projected/e5f10566-2d90-4c76-846b-8114279ad692-kube-api-access-4w27z\") pod \"service-ca-operator-5d46d9ffd4-fpx7q\" (UID: \"e5f10566-2d90-4c76-846b-8114279ad692\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-fpx7q" Jul 08 19:04:22.202246 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.202150 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9f2932e3-5510-44d8-bd92-76ebdac8ce3f-snapshots\") pod \"insights-operator-5c847dcfc4-l4fvs\" (UID: \"9f2932e3-5510-44d8-bd92-76ebdac8ce3f\") " pod="openshift-insights/insights-operator-5c847dcfc4-l4fvs" Jul 08 19:04:22.202246 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.202187 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9lrw\" (UniqueName: \"kubernetes.io/projected/13993822-93cd-4868-b430-faf701650fee-kube-api-access-j9lrw\") pod \"network-check-source-989f8b8dc-s5dzj\" (UID: \"13993822-93cd-4868-b430-faf701650fee\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-s5dzj" Jul 08 19:04:22.202246 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.202239 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/58ce9cbf-4ed4-4482-8786-d92dd2f9a0e0-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-7qzsf\" (UID: \"58ce9cbf-4ed4-4482-8786-d92dd2f9a0e0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-7qzsf" Jul 08 19:04:22.202572 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.202268 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/812854b5-4391-45db-b96d-cddb09c0d324-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-vvggh\" (UID: \"812854b5-4391-45db-b96d-cddb09c0d324\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vvggh" Jul 08 19:04:22.202572 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.202300 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f2932e3-5510-44d8-bd92-76ebdac8ce3f-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-l4fvs\" (UID: \"9f2932e3-5510-44d8-bd92-76ebdac8ce3f\") " pod="openshift-insights/insights-operator-5c847dcfc4-l4fvs" Jul 08 19:04:22.204078 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.202946 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e5f10566-2d90-4c76-846b-8114279ad692-config\") pod \"service-ca-operator-5d46d9ffd4-fpx7q\" (UID: \"e5f10566-2d90-4c76-846b-8114279ad692\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-fpx7q" Jul 08 19:04:22.205455 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.205431 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/58ce9cbf-4ed4-4482-8786-d92dd2f9a0e0-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-7qzsf\" (UID: \"58ce9cbf-4ed4-4482-8786-d92dd2f9a0e0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-7qzsf" Jul 08 19:04:22.205681 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.205665 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e5f10566-2d90-4c76-846b-8114279ad692-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-fpx7q\" (UID: \"e5f10566-2d90-4c76-846b-8114279ad692\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-fpx7q" Jul 08 19:04:22.206426 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.206406 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/58ce9cbf-4ed4-4482-8786-d92dd2f9a0e0-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-7qzsf\" (UID: \"58ce9cbf-4ed4-4482-8786-d92dd2f9a0e0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-7qzsf" Jul 08 19:04:22.211186 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.211161 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwjvg\" (UniqueName: \"kubernetes.io/projected/3fd49589-6210-4804-bccd-079d96770654-kube-api-access-bwjvg\") pod \"volume-data-source-validator-5d456fdbcd-69gjd\" (UID: \"3fd49589-6210-4804-bccd-079d96770654\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-69gjd" Jul 08 19:04:22.212239 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.212219 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w27z\" (UniqueName: \"kubernetes.io/projected/e5f10566-2d90-4c76-846b-8114279ad692-kube-api-access-4w27z\") pod \"service-ca-operator-5d46d9ffd4-fpx7q\" (UID: \"e5f10566-2d90-4c76-846b-8114279ad692\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-fpx7q" Jul 08 19:04:22.212437 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.212420 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2xdq\" (UniqueName: \"kubernetes.io/projected/58ce9cbf-4ed4-4482-8786-d92dd2f9a0e0-kube-api-access-q2xdq\") pod \"kube-storage-version-migrator-operator-846bf668f7-7qzsf\" (UID: \"58ce9cbf-4ed4-4482-8786-d92dd2f9a0e0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-7qzsf" Jul 08 19:04:22.218079 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.218059 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vvggh"] Jul 08 19:04:22.218168 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.218103 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-s5dzj"] Jul 08 19:04:22.218168 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.218126 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7556c5577f-p4h6b"] Jul 08 19:04:22.218168 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.218144 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-r7czv"] Jul 08 19:04:22.218279 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.218193 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-r7czv" Jul 08 19:04:22.221212 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.221192 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 08 19:04:22.222314 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.222293 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 08 19:04:22.222389 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.222314 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 08 19:04:22.222389 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.222350 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 08 19:04:22.222649 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.222630 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-lf8n5\"" Jul 08 19:04:22.238856 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.238827 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-54hnr" Jul 08 19:04:22.303387 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.303350 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9f2932e3-5510-44d8-bd92-76ebdac8ce3f-snapshots\") pod \"insights-operator-5c847dcfc4-l4fvs\" (UID: \"9f2932e3-5510-44d8-bd92-76ebdac8ce3f\") " pod="openshift-insights/insights-operator-5c847dcfc4-l4fvs" Jul 08 19:04:22.303592 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.303400 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9lrw\" (UniqueName: \"kubernetes.io/projected/13993822-93cd-4868-b430-faf701650fee-kube-api-access-j9lrw\") pod \"network-check-source-989f8b8dc-s5dzj\" (UID: \"13993822-93cd-4868-b430-faf701650fee\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-s5dzj" Jul 08 19:04:22.303592 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.303438 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/18b01b38-397e-492c-80c1-61217e74736f-stats-auth\") pod \"router-default-7556c5577f-p4h6b\" (UID: \"18b01b38-397e-492c-80c1-61217e74736f\") " pod="openshift-ingress/router-default-7556c5577f-p4h6b" Jul 08 19:04:22.303592 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.303455 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8b44\" (UniqueName: \"kubernetes.io/projected/18b01b38-397e-492c-80c1-61217e74736f-kube-api-access-j8b44\") pod \"router-default-7556c5577f-p4h6b\" (UID: \"18b01b38-397e-492c-80c1-61217e74736f\") " pod="openshift-ingress/router-default-7556c5577f-p4h6b" Jul 08 19:04:22.303592 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.303473 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18b01b38-397e-492c-80c1-61217e74736f-service-ca-bundle\") pod \"router-default-7556c5577f-p4h6b\" (UID: \"18b01b38-397e-492c-80c1-61217e74736f\") " pod="openshift-ingress/router-default-7556c5577f-p4h6b" Jul 08 19:04:22.303592 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.303499 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/812854b5-4391-45db-b96d-cddb09c0d324-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-vvggh\" (UID: \"812854b5-4391-45db-b96d-cddb09c0d324\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vvggh" Jul 08 19:04:22.303849 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.303617 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f2932e3-5510-44d8-bd92-76ebdac8ce3f-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-l4fvs\" (UID: \"9f2932e3-5510-44d8-bd92-76ebdac8ce3f\") " pod="openshift-insights/insights-operator-5c847dcfc4-l4fvs" Jul 08 19:04:22.303849 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.303673 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f2932e3-5510-44d8-bd92-76ebdac8ce3f-serving-cert\") pod \"insights-operator-5c847dcfc4-l4fvs\" (UID: \"9f2932e3-5510-44d8-bd92-76ebdac8ce3f\") " pod="openshift-insights/insights-operator-5c847dcfc4-l4fvs" Jul 08 19:04:22.303849 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.303697 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9cebd65f-6a73-46e8-9995-7797c67acb53-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-r7czv\" (UID: \"9cebd65f-6a73-46e8-9995-7797c67acb53\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-r7czv" Jul 08 19:04:22.303849 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.303727 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/18b01b38-397e-492c-80c1-61217e74736f-default-certificate\") pod \"router-default-7556c5577f-p4h6b\" (UID: \"18b01b38-397e-492c-80c1-61217e74736f\") " pod="openshift-ingress/router-default-7556c5577f-p4h6b" Jul 08 19:04:22.303849 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.303770 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w49k\" (UniqueName: \"kubernetes.io/projected/812854b5-4391-45db-b96d-cddb09c0d324-kube-api-access-8w49k\") pod \"cluster-samples-operator-56fcf86bf-vvggh\" (UID: \"812854b5-4391-45db-b96d-cddb09c0d324\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vvggh" Jul 08 19:04:22.303849 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.303818 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f2932e3-5510-44d8-bd92-76ebdac8ce3f-tmp\") pod \"insights-operator-5c847dcfc4-l4fvs\" (UID: \"9f2932e3-5510-44d8-bd92-76ebdac8ce3f\") " pod="openshift-insights/insights-operator-5c847dcfc4-l4fvs" Jul 08 19:04:22.303849 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.303840 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f2932e3-5510-44d8-bd92-76ebdac8ce3f-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-l4fvs\" (UID: \"9f2932e3-5510-44d8-bd92-76ebdac8ce3f\") " pod="openshift-insights/insights-operator-5c847dcfc4-l4fvs" Jul 08 19:04:22.304192 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.303859 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18b01b38-397e-492c-80c1-61217e74736f-metrics-certs\") pod \"router-default-7556c5577f-p4h6b\" (UID: \"18b01b38-397e-492c-80c1-61217e74736f\") " pod="openshift-ingress/router-default-7556c5577f-p4h6b" Jul 08 19:04:22.304192 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.303884 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9cebd65f-6a73-46e8-9995-7797c67acb53-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-r7czv\" (UID: \"9cebd65f-6a73-46e8-9995-7797c67acb53\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-r7czv" Jul 08 19:04:22.304192 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.303912 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhfm2\" (UniqueName: \"kubernetes.io/projected/9f2932e3-5510-44d8-bd92-76ebdac8ce3f-kube-api-access-fhfm2\") pod \"insights-operator-5c847dcfc4-l4fvs\" (UID: \"9f2932e3-5510-44d8-bd92-76ebdac8ce3f\") " pod="openshift-insights/insights-operator-5c847dcfc4-l4fvs" Jul 08 19:04:22.304192 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.303936 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hwk9\" (UniqueName: \"kubernetes.io/projected/9cebd65f-6a73-46e8-9995-7797c67acb53-kube-api-access-5hwk9\") pod \"cluster-monitoring-operator-64786d9cbb-r7czv\" (UID: \"9cebd65f-6a73-46e8-9995-7797c67acb53\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-r7czv" Jul 08 19:04:22.304192 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.304118 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9f2932e3-5510-44d8-bd92-76ebdac8ce3f-snapshots\") pod \"insights-operator-5c847dcfc4-l4fvs\" (UID: \"9f2932e3-5510-44d8-bd92-76ebdac8ce3f\") " pod="openshift-insights/insights-operator-5c847dcfc4-l4fvs" Jul 08 19:04:22.304427 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.304279 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f2932e3-5510-44d8-bd92-76ebdac8ce3f-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-l4fvs\" (UID: \"9f2932e3-5510-44d8-bd92-76ebdac8ce3f\") " pod="openshift-insights/insights-operator-5c847dcfc4-l4fvs" Jul 08 19:04:22.304482 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:22.304429 2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 19:04:22.304482 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.304447 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f2932e3-5510-44d8-bd92-76ebdac8ce3f-tmp\") pod \"insights-operator-5c847dcfc4-l4fvs\" (UID: \"9f2932e3-5510-44d8-bd92-76ebdac8ce3f\") " pod="openshift-insights/insights-operator-5c847dcfc4-l4fvs" Jul 08 19:04:22.304566 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:22.304514 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/812854b5-4391-45db-b96d-cddb09c0d324-samples-operator-tls podName:812854b5-4391-45db-b96d-cddb09c0d324 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:22.804493637 +0000 UTC m=+46.063446557 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/812854b5-4391-45db-b96d-cddb09c0d324-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-vvggh" (UID: "812854b5-4391-45db-b96d-cddb09c0d324") : secret "samples-operator-tls" not found Jul 08 19:04:22.305124 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.305093 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f2932e3-5510-44d8-bd92-76ebdac8ce3f-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-l4fvs\" (UID: \"9f2932e3-5510-44d8-bd92-76ebdac8ce3f\") " pod="openshift-insights/insights-operator-5c847dcfc4-l4fvs" Jul 08 19:04:22.307012 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.306988 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f2932e3-5510-44d8-bd92-76ebdac8ce3f-serving-cert\") pod \"insights-operator-5c847dcfc4-l4fvs\" (UID: \"9f2932e3-5510-44d8-bd92-76ebdac8ce3f\") " pod="openshift-insights/insights-operator-5c847dcfc4-l4fvs" Jul 08 19:04:22.319143 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.319111 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:22.320439 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.320409 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhfm2\" (UniqueName: \"kubernetes.io/projected/9f2932e3-5510-44d8-bd92-76ebdac8ce3f-kube-api-access-fhfm2\") pod \"insights-operator-5c847dcfc4-l4fvs\" (UID: \"9f2932e3-5510-44d8-bd92-76ebdac8ce3f\") " pod="openshift-insights/insights-operator-5c847dcfc4-l4fvs" Jul 08 19:04:22.321968 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.321914 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9lrw\" (UniqueName: \"kubernetes.io/projected/13993822-93cd-4868-b430-faf701650fee-kube-api-access-j9lrw\") pod \"network-check-source-989f8b8dc-s5dzj\" (UID: \"13993822-93cd-4868-b430-faf701650fee\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-s5dzj" Jul 08 19:04:22.322339 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.322321 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w49k\" (UniqueName: \"kubernetes.io/projected/812854b5-4391-45db-b96d-cddb09c0d324-kube-api-access-8w49k\") pod \"cluster-samples-operator-56fcf86bf-vvggh\" (UID: \"812854b5-4391-45db-b96d-cddb09c0d324\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vvggh" Jul 08 19:04:22.341155 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.341124 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-fpx7q" Jul 08 19:04:22.358884 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.358856 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-7qzsf" Jul 08 19:04:22.383276 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.383084 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-69gjd" Jul 08 19:04:22.398702 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.398675 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-54hnr"] Jul 08 19:04:22.401193 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:04:22.401164 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ebcfaa0_2d87_4a85_9e2e_13d4e230e39a.slice/crio-21be8673dc9a62a151c753ad8cfff39a3a3447bb43afefd84aa3b63f7c18e2e2 WatchSource:0}: Error finding container 21be8673dc9a62a151c753ad8cfff39a3a3447bb43afefd84aa3b63f7c18e2e2: Status 404 returned error can't find the container with id 21be8673dc9a62a151c753ad8cfff39a3a3447bb43afefd84aa3b63f7c18e2e2 Jul 08 19:04:22.404455 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.404429 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18b01b38-397e-492c-80c1-61217e74736f-metrics-certs\") pod \"router-default-7556c5577f-p4h6b\" (UID: \"18b01b38-397e-492c-80c1-61217e74736f\") " pod="openshift-ingress/router-default-7556c5577f-p4h6b" Jul 08 19:04:22.404538 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.404485 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9cebd65f-6a73-46e8-9995-7797c67acb53-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-r7czv\" (UID: \"9cebd65f-6a73-46e8-9995-7797c67acb53\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-r7czv" Jul 08 19:04:22.404538 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.404518 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hwk9\" (UniqueName: \"kubernetes.io/projected/9cebd65f-6a73-46e8-9995-7797c67acb53-kube-api-access-5hwk9\") pod \"cluster-monitoring-operator-64786d9cbb-r7czv\" (UID: \"9cebd65f-6a73-46e8-9995-7797c67acb53\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-r7czv" Jul 08 19:04:22.404689 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.404624 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/18b01b38-397e-492c-80c1-61217e74736f-stats-auth\") pod \"router-default-7556c5577f-p4h6b\" (UID: \"18b01b38-397e-492c-80c1-61217e74736f\") " pod="openshift-ingress/router-default-7556c5577f-p4h6b" Jul 08 19:04:22.404689 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.404651 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8b44\" (UniqueName: \"kubernetes.io/projected/18b01b38-397e-492c-80c1-61217e74736f-kube-api-access-j8b44\") pod \"router-default-7556c5577f-p4h6b\" (UID: \"18b01b38-397e-492c-80c1-61217e74736f\") " pod="openshift-ingress/router-default-7556c5577f-p4h6b" Jul 08 19:04:22.404689 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.404678 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18b01b38-397e-492c-80c1-61217e74736f-service-ca-bundle\") pod \"router-default-7556c5577f-p4h6b\" (UID: \"18b01b38-397e-492c-80c1-61217e74736f\") " pod="openshift-ingress/router-default-7556c5577f-p4h6b" Jul 08 19:04:22.404840 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:22.404814 2558 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 19:04:22.404895 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:22.404880 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cebd65f-6a73-46e8-9995-7797c67acb53-cluster-monitoring-operator-tls podName:9cebd65f-6a73-46e8-9995-7797c67acb53 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:22.904859872 +0000 UTC m=+46.163812777 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9cebd65f-6a73-46e8-9995-7797c67acb53-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-r7czv" (UID: "9cebd65f-6a73-46e8-9995-7797c67acb53") : secret "cluster-monitoring-operator-tls" not found Jul 08 19:04:22.405053 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:22.405036 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18b01b38-397e-492c-80c1-61217e74736f-service-ca-bundle podName:18b01b38-397e-492c-80c1-61217e74736f nodeName:}" failed. No retries permitted until 2026-07-08 19:04:22.905018333 +0000 UTC m=+46.163971254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/18b01b38-397e-492c-80c1-61217e74736f-service-ca-bundle") pod "router-default-7556c5577f-p4h6b" (UID: "18b01b38-397e-492c-80c1-61217e74736f") : configmap references non-existent config key: service-ca.crt Jul 08 19:04:22.405140 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:22.405105 2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 19:04:22.405140 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:22.405140 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18b01b38-397e-492c-80c1-61217e74736f-metrics-certs podName:18b01b38-397e-492c-80c1-61217e74736f nodeName:}" failed. No retries permitted until 2026-07-08 19:04:22.905128466 +0000 UTC m=+46.164081370 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18b01b38-397e-492c-80c1-61217e74736f-metrics-certs") pod "router-default-7556c5577f-p4h6b" (UID: "18b01b38-397e-492c-80c1-61217e74736f") : secret "router-metrics-certs-default" not found Jul 08 19:04:22.405247 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.405187 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9cebd65f-6a73-46e8-9995-7797c67acb53-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-r7czv\" (UID: \"9cebd65f-6a73-46e8-9995-7797c67acb53\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-r7czv" Jul 08 19:04:22.405247 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.405227 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/18b01b38-397e-492c-80c1-61217e74736f-default-certificate\") pod \"router-default-7556c5577f-p4h6b\" (UID: \"18b01b38-397e-492c-80c1-61217e74736f\") " pod="openshift-ingress/router-default-7556c5577f-p4h6b" Jul 08 19:04:22.405330 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.405249 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-l4fvs" Jul 08 19:04:22.406337 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.406291 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9cebd65f-6a73-46e8-9995-7797c67acb53-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-r7czv\" (UID: \"9cebd65f-6a73-46e8-9995-7797c67acb53\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-r7czv" Jul 08 19:04:22.408807 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.408510 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/18b01b38-397e-492c-80c1-61217e74736f-stats-auth\") pod \"router-default-7556c5577f-p4h6b\" (UID: \"18b01b38-397e-492c-80c1-61217e74736f\") " pod="openshift-ingress/router-default-7556c5577f-p4h6b" Jul 08 19:04:22.409233 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.409083 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:22.409380 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.409362 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/18b01b38-397e-492c-80c1-61217e74736f-default-certificate\") pod \"router-default-7556c5577f-p4h6b\" (UID: \"18b01b38-397e-492c-80c1-61217e74736f\") " pod="openshift-ingress/router-default-7556c5577f-p4h6b" Jul 08 19:04:22.416968 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.414528 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8b44\" (UniqueName: \"kubernetes.io/projected/18b01b38-397e-492c-80c1-61217e74736f-kube-api-access-j8b44\") pod \"router-default-7556c5577f-p4h6b\" (UID: \"18b01b38-397e-492c-80c1-61217e74736f\") " pod="openshift-ingress/router-default-7556c5577f-p4h6b" Jul 08 19:04:22.416968 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.415395 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hwk9\" (UniqueName: \"kubernetes.io/projected/9cebd65f-6a73-46e8-9995-7797c67acb53-kube-api-access-5hwk9\") pod \"cluster-monitoring-operator-64786d9cbb-r7czv\" (UID: \"9cebd65f-6a73-46e8-9995-7797c67acb53\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-r7czv" Jul 08 19:04:22.479004 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.478786 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-s5dzj" Jul 08 19:04:22.480435 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.480401 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-54hnr" event={"ID":"1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a","Type":"ContainerStarted","Data":"21be8673dc9a62a151c753ad8cfff39a3a3447bb43afefd84aa3b63f7c18e2e2"} Jul 08 19:04:22.515152 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.514869 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-fpx7q"] Jul 08 19:04:22.521722 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:04:22.521661 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5f10566_2d90_4c76_846b_8114279ad692.slice/crio-3e3c34dff98384d2f93f853b4055515bc117a93a791d7f39ae84c92a1325f6f0 WatchSource:0}: Error finding container 3e3c34dff98384d2f93f853b4055515bc117a93a791d7f39ae84c92a1325f6f0: Status 404 returned error can't find the container with id 3e3c34dff98384d2f93f853b4055515bc117a93a791d7f39ae84c92a1325f6f0 Jul 08 19:04:22.536715 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.536664 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-7qzsf"] Jul 08 19:04:22.542391 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:04:22.542367 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58ce9cbf_4ed4_4482_8786_d92dd2f9a0e0.slice/crio-7a78edc9811086aa629ca98df1e6e1eef8cbd931d68f46feded1bbbeb2de7dbe WatchSource:0}: Error finding container 7a78edc9811086aa629ca98df1e6e1eef8cbd931d68f46feded1bbbeb2de7dbe: Status 404 returned error can't find the container with id 7a78edc9811086aa629ca98df1e6e1eef8cbd931d68f46feded1bbbeb2de7dbe Jul 08 19:04:22.561345 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.561314 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-69gjd"] Jul 08 19:04:22.567150 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:04:22.567093 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fd49589_6210_4804_bccd_079d96770654.slice/crio-f0b38f217946385d1792bffdc7adf6e4040b2c3ee8ed4fb467f9e92e2ae6a096 WatchSource:0}: Error finding container f0b38f217946385d1792bffdc7adf6e4040b2c3ee8ed4fb467f9e92e2ae6a096: Status 404 returned error can't find the container with id f0b38f217946385d1792bffdc7adf6e4040b2c3ee8ed4fb467f9e92e2ae6a096 Jul 08 19:04:22.585398 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.585377 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-l4fvs"] Jul 08 19:04:22.587894 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:04:22.587868 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f2932e3_5510_44d8_bd92_76ebdac8ce3f.slice/crio-0f4b8895296839ccc7eaa695c3b738e865895e6a68c0f0854738005dce65c755 WatchSource:0}: Error finding container 0f4b8895296839ccc7eaa695c3b738e865895e6a68c0f0854738005dce65c755: Status 404 returned error can't find the container with id 0f4b8895296839ccc7eaa695c3b738e865895e6a68c0f0854738005dce65c755 Jul 08 19:04:22.640046 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.639984 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-s5dzj"] Jul 08 19:04:22.642316 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:04:22.642293 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13993822_93cd_4868_b430_faf701650fee.slice/crio-a81468ce070ca484847b367b019453f9bb6a98139a3531e285bffa538526abb4 WatchSource:0}: Error finding container a81468ce070ca484847b367b019453f9bb6a98139a3531e285bffa538526abb4: Status 404 returned error can't find the container with id a81468ce070ca484847b367b019453f9bb6a98139a3531e285bffa538526abb4 Jul 08 19:04:22.809370 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.809285 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/812854b5-4391-45db-b96d-cddb09c0d324-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-vvggh\" (UID: \"812854b5-4391-45db-b96d-cddb09c0d324\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vvggh" Jul 08 19:04:22.809370 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.809348 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-registry-tls\") pod \"image-registry-7c4688bdd8-xlx9p\" (UID: \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\") " pod="openshift-image-registry/image-registry-7c4688bdd8-xlx9p" Jul 08 19:04:22.809600 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:22.809430 2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 19:04:22.809600 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:22.809478 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 19:04:22.809600 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:22.809490 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/812854b5-4391-45db-b96d-cddb09c0d324-samples-operator-tls podName:812854b5-4391-45db-b96d-cddb09c0d324 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:23.809476229 +0000 UTC m=+47.068429131 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/812854b5-4391-45db-b96d-cddb09c0d324-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-vvggh" (UID: "812854b5-4391-45db-b96d-cddb09c0d324") : secret "samples-operator-tls" not found Jul 08 19:04:22.809600 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:22.809493 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c4688bdd8-xlx9p: secret "image-registry-tls" not found Jul 08 19:04:22.809600 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:22.809541 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-registry-tls podName:fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:30.809525236 +0000 UTC m=+54.068478161 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-registry-tls") pod "image-registry-7c4688bdd8-xlx9p" (UID: "fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7") : secret "image-registry-tls" not found Jul 08 19:04:22.910562 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.910531 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eae871c8-0143-47c6-853c-2413d8f6fa70-metrics-tls\") pod \"dns-default-ndtk5\" (UID: \"eae871c8-0143-47c6-853c-2413d8f6fa70\") " pod="openshift-dns/dns-default-ndtk5" Jul 08 19:04:22.911198 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.910569 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18b01b38-397e-492c-80c1-61217e74736f-metrics-certs\") pod \"router-default-7556c5577f-p4h6b\" (UID: \"18b01b38-397e-492c-80c1-61217e74736f\") " pod="openshift-ingress/router-default-7556c5577f-p4h6b" Jul 08 19:04:22.911198 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.910617 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9cebd65f-6a73-46e8-9995-7797c67acb53-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-r7czv\" (UID: \"9cebd65f-6a73-46e8-9995-7797c67acb53\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-r7czv" Jul 08 19:04:22.911198 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:22.910685 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 19:04:22.911198 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:22.910690 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18b01b38-397e-492c-80c1-61217e74736f-service-ca-bundle\") pod \"router-default-7556c5577f-p4h6b\" (UID: \"18b01b38-397e-492c-80c1-61217e74736f\") " pod="openshift-ingress/router-default-7556c5577f-p4h6b" Jul 08 19:04:22.911198 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:22.910744 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eae871c8-0143-47c6-853c-2413d8f6fa70-metrics-tls podName:eae871c8-0143-47c6-853c-2413d8f6fa70 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:30.910726695 +0000 UTC m=+54.169679618 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eae871c8-0143-47c6-853c-2413d8f6fa70-metrics-tls") pod "dns-default-ndtk5" (UID: "eae871c8-0143-47c6-853c-2413d8f6fa70") : secret "dns-default-metrics-tls" not found Jul 08 19:04:22.911198 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:22.910750 2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 19:04:22.911198 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:22.910772 2558 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 19:04:22.911198 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:22.910790 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18b01b38-397e-492c-80c1-61217e74736f-service-ca-bundle podName:18b01b38-397e-492c-80c1-61217e74736f nodeName:}" failed. No retries permitted until 2026-07-08 19:04:23.910780279 +0000 UTC m=+47.169733195 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/18b01b38-397e-492c-80c1-61217e74736f-service-ca-bundle") pod "router-default-7556c5577f-p4h6b" (UID: "18b01b38-397e-492c-80c1-61217e74736f") : configmap references non-existent config key: service-ca.crt Jul 08 19:04:22.911198 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:22.910807 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18b01b38-397e-492c-80c1-61217e74736f-metrics-certs podName:18b01b38-397e-492c-80c1-61217e74736f nodeName:}" failed. No retries permitted until 2026-07-08 19:04:23.910798999 +0000 UTC m=+47.169751902 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18b01b38-397e-492c-80c1-61217e74736f-metrics-certs") pod "router-default-7556c5577f-p4h6b" (UID: "18b01b38-397e-492c-80c1-61217e74736f") : secret "router-metrics-certs-default" not found Jul 08 19:04:22.911198 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:22.910821 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cebd65f-6a73-46e8-9995-7797c67acb53-cluster-monitoring-operator-tls podName:9cebd65f-6a73-46e8-9995-7797c67acb53 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:23.910813081 +0000 UTC m=+47.169765989 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9cebd65f-6a73-46e8-9995-7797c67acb53-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-r7czv" (UID: "9cebd65f-6a73-46e8-9995-7797c67acb53") : secret "cluster-monitoring-operator-tls" not found Jul 08 19:04:23.485241 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:23.485164 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-69gjd" event={"ID":"3fd49589-6210-4804-bccd-079d96770654","Type":"ContainerStarted","Data":"f0b38f217946385d1792bffdc7adf6e4040b2c3ee8ed4fb467f9e92e2ae6a096"} Jul 08 19:04:23.486635 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:23.486553 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-7qzsf" event={"ID":"58ce9cbf-4ed4-4482-8786-d92dd2f9a0e0","Type":"ContainerStarted","Data":"7a78edc9811086aa629ca98df1e6e1eef8cbd931d68f46feded1bbbeb2de7dbe"} Jul 08 19:04:23.488130 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:23.488072 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-s5dzj" event={"ID":"13993822-93cd-4868-b430-faf701650fee","Type":"ContainerStarted","Data":"a81468ce070ca484847b367b019453f9bb6a98139a3531e285bffa538526abb4"} Jul 08 19:04:23.489824 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:23.489774 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-l4fvs" event={"ID":"9f2932e3-5510-44d8-bd92-76ebdac8ce3f","Type":"ContainerStarted","Data":"0f4b8895296839ccc7eaa695c3b738e865895e6a68c0f0854738005dce65c755"} Jul 08 19:04:23.491285 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:23.491235 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-fpx7q" event={"ID":"e5f10566-2d90-4c76-846b-8114279ad692","Type":"ContainerStarted","Data":"3e3c34dff98384d2f93f853b4055515bc117a93a791d7f39ae84c92a1325f6f0"} Jul 08 19:04:23.820032 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:23.819949 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/812854b5-4391-45db-b96d-cddb09c0d324-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-vvggh\" (UID: \"812854b5-4391-45db-b96d-cddb09c0d324\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vvggh" Jul 08 19:04:23.820235 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:23.820216 2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 19:04:23.820298 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:23.820290 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/812854b5-4391-45db-b96d-cddb09c0d324-samples-operator-tls podName:812854b5-4391-45db-b96d-cddb09c0d324 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:25.820270534 +0000 UTC m=+49.079223443 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/812854b5-4391-45db-b96d-cddb09c0d324-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-vvggh" (UID: "812854b5-4391-45db-b96d-cddb09c0d324") : secret "samples-operator-tls" not found Jul 08 19:04:23.921622 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:23.920756 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18b01b38-397e-492c-80c1-61217e74736f-metrics-certs\") pod \"router-default-7556c5577f-p4h6b\" (UID: \"18b01b38-397e-492c-80c1-61217e74736f\") " pod="openshift-ingress/router-default-7556c5577f-p4h6b" Jul 08 19:04:23.921622 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:23.920816 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9cebd65f-6a73-46e8-9995-7797c67acb53-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-r7czv\" (UID: \"9cebd65f-6a73-46e8-9995-7797c67acb53\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-r7czv" Jul 08 19:04:23.921622 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:23.920910 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18b01b38-397e-492c-80c1-61217e74736f-service-ca-bundle\") pod \"router-default-7556c5577f-p4h6b\" (UID: \"18b01b38-397e-492c-80c1-61217e74736f\") " pod="openshift-ingress/router-default-7556c5577f-p4h6b" Jul 08 19:04:23.921622 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:23.921093 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18b01b38-397e-492c-80c1-61217e74736f-service-ca-bundle podName:18b01b38-397e-492c-80c1-61217e74736f nodeName:}" failed. No retries permitted until 2026-07-08 19:04:25.921074806 +0000 UTC m=+49.180027722 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/18b01b38-397e-492c-80c1-61217e74736f-service-ca-bundle") pod "router-default-7556c5577f-p4h6b" (UID: "18b01b38-397e-492c-80c1-61217e74736f") : configmap references non-existent config key: service-ca.crt Jul 08 19:04:23.921622 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:23.921513 2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 19:04:23.921622 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:23.921589 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18b01b38-397e-492c-80c1-61217e74736f-metrics-certs podName:18b01b38-397e-492c-80c1-61217e74736f nodeName:}" failed. No retries permitted until 2026-07-08 19:04:25.921554344 +0000 UTC m=+49.180507249 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18b01b38-397e-492c-80c1-61217e74736f-metrics-certs") pod "router-default-7556c5577f-p4h6b" (UID: "18b01b38-397e-492c-80c1-61217e74736f") : secret "router-metrics-certs-default" not found Jul 08 19:04:23.922401 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:23.921660 2558 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 19:04:23.922401 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:23.921746 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cebd65f-6a73-46e8-9995-7797c67acb53-cluster-monitoring-operator-tls podName:9cebd65f-6a73-46e8-9995-7797c67acb53 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:25.921729826 +0000 UTC m=+49.180682731 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9cebd65f-6a73-46e8-9995-7797c67acb53-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-r7czv" (UID: "9cebd65f-6a73-46e8-9995-7797c67acb53") : secret "cluster-monitoring-operator-tls" not found Jul 08 19:04:24.679481 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:24.679395 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:25.499712 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:25.499677 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-69gjd" event={"ID":"3fd49589-6210-4804-bccd-079d96770654","Type":"ContainerStarted","Data":"f374fc8b42d3df0b7b917131049539c7ea486302369a483848e9d16863fd9d64"} Jul 08 19:04:25.520306 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:25.520252 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-69gjd" podStartSLOduration=2.46170036 podStartE2EDuration="4.520235886s" totalImagesPullingTime="2.058535526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:04:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:04:25.519329566 +0000 UTC m=+48.778282509" watchObservedRunningTime="2026-07-08 19:04:25.520235886 +0000 UTC m=+48.779188810" Jul 08 19:04:25.626569 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:25.626519 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:25.727753 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:25.727714 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:25.841031 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:25.840991 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/812854b5-4391-45db-b96d-cddb09c0d324-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-vvggh\" (UID: \"812854b5-4391-45db-b96d-cddb09c0d324\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vvggh" Jul 08 19:04:25.841185 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:25.841031 2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 19:04:25.841185 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:25.841105 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/812854b5-4391-45db-b96d-cddb09c0d324-samples-operator-tls podName:812854b5-4391-45db-b96d-cddb09c0d324 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:29.841085302 +0000 UTC m=+53.100038206 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/812854b5-4391-45db-b96d-cddb09c0d324-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-vvggh" (UID: "812854b5-4391-45db-b96d-cddb09c0d324") : secret "samples-operator-tls" not found Jul 08 19:04:25.941823 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:25.941525 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18b01b38-397e-492c-80c1-61217e74736f-metrics-certs\") pod \"router-default-7556c5577f-p4h6b\" (UID: \"18b01b38-397e-492c-80c1-61217e74736f\") " pod="openshift-ingress/router-default-7556c5577f-p4h6b" Jul 08 19:04:25.941823 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:25.941590 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9cebd65f-6a73-46e8-9995-7797c67acb53-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-r7czv\" (UID: \"9cebd65f-6a73-46e8-9995-7797c67acb53\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-r7czv" Jul 08 19:04:25.941823 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:25.941679 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18b01b38-397e-492c-80c1-61217e74736f-service-ca-bundle\") pod \"router-default-7556c5577f-p4h6b\" (UID: \"18b01b38-397e-492c-80c1-61217e74736f\") " pod="openshift-ingress/router-default-7556c5577f-p4h6b" Jul 08 19:04:25.941823 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:25.941688 2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 19:04:25.941823 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:25.941762 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18b01b38-397e-492c-80c1-61217e74736f-metrics-certs podName:18b01b38-397e-492c-80c1-61217e74736f nodeName:}" failed. No retries permitted until 2026-07-08 19:04:29.941731556 +0000 UTC m=+53.200684471 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18b01b38-397e-492c-80c1-61217e74736f-metrics-certs") pod "router-default-7556c5577f-p4h6b" (UID: "18b01b38-397e-492c-80c1-61217e74736f") : secret "router-metrics-certs-default" not found Jul 08 19:04:25.941823 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:25.941766 2558 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 19:04:25.942239 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:25.941820 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cebd65f-6a73-46e8-9995-7797c67acb53-cluster-monitoring-operator-tls podName:9cebd65f-6a73-46e8-9995-7797c67acb53 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:29.941803432 +0000 UTC m=+53.200756352 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9cebd65f-6a73-46e8-9995-7797c67acb53-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-r7czv" (UID: "9cebd65f-6a73-46e8-9995-7797c67acb53") : secret "cluster-monitoring-operator-tls" not found Jul 08 19:04:25.942239 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:25.941852 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18b01b38-397e-492c-80c1-61217e74736f-service-ca-bundle podName:18b01b38-397e-492c-80c1-61217e74736f nodeName:}" failed. No retries permitted until 2026-07-08 19:04:29.94184449 +0000 UTC m=+53.200797395 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/18b01b38-397e-492c-80c1-61217e74736f-service-ca-bundle") pod "router-default-7556c5577f-p4h6b" (UID: "18b01b38-397e-492c-80c1-61217e74736f") : configmap references non-existent config key: service-ca.crt Jul 08 19:04:26.517567 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:26.517535 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-pr89r"] Jul 08 19:04:26.541159 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:26.541131 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-pr89r"] Jul 08 19:04:26.541310 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:26.541220 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-pr89r" Jul 08 19:04:26.544040 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:26.544010 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 08 19:04:26.544165 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:26.544047 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 08 19:04:26.545122 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:26.545103 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-cktkj\"" Jul 08 19:04:26.647854 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:26.647787 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a06e8e9a-6466-46ce-acaa-abdda774607d-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-pr89r\" (UID: \"a06e8e9a-6466-46ce-acaa-abdda774607d\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-pr89r" Jul 08 19:04:26.648056 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:26.648032 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a06e8e9a-6466-46ce-acaa-abdda774607d-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-pr89r\" (UID: \"a06e8e9a-6466-46ce-acaa-abdda774607d\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-pr89r" Jul 08 19:04:26.664718 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:26.664678 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:26.745865 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:26.745821 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:26.748649 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:26.748627 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a06e8e9a-6466-46ce-acaa-abdda774607d-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-pr89r\" (UID: \"a06e8e9a-6466-46ce-acaa-abdda774607d\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-pr89r" Jul 08 19:04:26.748825 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:26.748808 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a06e8e9a-6466-46ce-acaa-abdda774607d-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-pr89r\" (UID: \"a06e8e9a-6466-46ce-acaa-abdda774607d\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-pr89r" Jul 08 19:04:26.748970 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:26.748953 2558 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 19:04:26.749022 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:26.749014 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a06e8e9a-6466-46ce-acaa-abdda774607d-networking-console-plugin-cert podName:a06e8e9a-6466-46ce-acaa-abdda774607d nodeName:}" failed. No retries permitted until 2026-07-08 19:04:27.248999506 +0000 UTC m=+50.507952409 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a06e8e9a-6466-46ce-acaa-abdda774607d-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-pr89r" (UID: "a06e8e9a-6466-46ce-acaa-abdda774607d") : secret "networking-console-plugin-cert" not found Jul 08 19:04:26.749223 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:26.749207 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a06e8e9a-6466-46ce-acaa-abdda774607d-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-pr89r\" (UID: \"a06e8e9a-6466-46ce-acaa-abdda774607d\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-pr89r" Jul 08 19:04:26.811008 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:26.810938 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:26.911880 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:26.911841 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:27.012866 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:27.012832 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:27.085947 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:27.085869 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:27.252623 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:27.252589 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a06e8e9a-6466-46ce-acaa-abdda774607d-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-pr89r\" (UID: \"a06e8e9a-6466-46ce-acaa-abdda774607d\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-pr89r" Jul 08 19:04:27.252793 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:27.252724 2558 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 19:04:27.252793 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:27.252781 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a06e8e9a-6466-46ce-acaa-abdda774607d-networking-console-plugin-cert podName:a06e8e9a-6466-46ce-acaa-abdda774607d nodeName:}" failed. No retries permitted until 2026-07-08 19:04:28.252766928 +0000 UTC m=+51.511719850 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a06e8e9a-6466-46ce-acaa-abdda774607d-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-pr89r" (UID: "a06e8e9a-6466-46ce-acaa-abdda774607d") : secret "networking-console-plugin-cert" not found Jul 08 19:04:28.262331 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:28.262277 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a06e8e9a-6466-46ce-acaa-abdda774607d-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-pr89r\" (UID: \"a06e8e9a-6466-46ce-acaa-abdda774607d\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-pr89r" Jul 08 19:04:28.262759 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:28.262442 2558 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 19:04:28.262759 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:28.262508 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a06e8e9a-6466-46ce-acaa-abdda774607d-networking-console-plugin-cert podName:a06e8e9a-6466-46ce-acaa-abdda774607d nodeName:}" failed. No retries permitted until 2026-07-08 19:04:30.262491809 +0000 UTC m=+53.521444712 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a06e8e9a-6466-46ce-acaa-abdda774607d-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-pr89r" (UID: "a06e8e9a-6466-46ce-acaa-abdda774607d") : secret "networking-console-plugin-cert" not found Jul 08 19:04:29.393003 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:29.392964 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:29.461675 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:29.461637 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:29.517065 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:29.517029 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-l4fvs" event={"ID":"9f2932e3-5510-44d8-bd92-76ebdac8ce3f","Type":"ContainerStarted","Data":"55b9910906c4d27556d04b274773c2e0e473b65382337a5c9ea362d7f289996b"} Jul 08 19:04:29.519678 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:29.519393 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-fpx7q" event={"ID":"e5f10566-2d90-4c76-846b-8114279ad692","Type":"ContainerStarted","Data":"3e356f0d89b3199deda45bf2d402a44e5069051442a48aa3b16a925e705bb9e7"} Jul 08 19:04:29.521262 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:29.521242 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-54hnr_1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a/console-operator/0.log" Jul 08 19:04:29.521342 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:29.521279 2558 generic.go:356] "Generic (PLEG): container finished" podID="1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a" containerID="73fb4178a72994765c40c8c17e84de7404bb47bd3d7581adb8489b2700290c1f" exitCode=255 Jul 08 19:04:29.521447 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:29.521344 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-54hnr" event={"ID":"1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a","Type":"ContainerDied","Data":"73fb4178a72994765c40c8c17e84de7404bb47bd3d7581adb8489b2700290c1f"} Jul 08 19:04:29.521637 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:29.521611 2558 scope.go:117] "RemoveContainer" containerID="73fb4178a72994765c40c8c17e84de7404bb47bd3d7581adb8489b2700290c1f" Jul 08 19:04:29.523245 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:29.523221 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-7qzsf" event={"ID":"58ce9cbf-4ed4-4482-8786-d92dd2f9a0e0","Type":"ContainerStarted","Data":"8ab04919eaae07e3e7c46a19f2f818dfcf41a57d211eb55bb74f4da26acc6968"} Jul 08 19:04:29.525442 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:29.525420 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-s5dzj" event={"ID":"13993822-93cd-4868-b430-faf701650fee","Type":"ContainerStarted","Data":"1b3f37f926bd687e76e028e65db08bcdaab114eded3795041ab53ca352f73f2f"} Jul 08 19:04:29.537493 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:29.537440 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-l4fvs" podStartSLOduration=2.384684782 podStartE2EDuration="8.537420306s" totalImagesPullingTime="6.152735524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:04:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:04:29.536440692 +0000 UTC m=+52.795393630" watchObservedRunningTime="2026-07-08 19:04:29.537420306 +0000 UTC m=+52.796373231" Jul 08 19:04:29.588099 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:29.588040 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-7qzsf" podStartSLOduration=2.39491872 podStartE2EDuration="8.588023974s" totalImagesPullingTime="6.193105254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:04:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:04:29.587621704 +0000 UTC m=+52.846574649" watchObservedRunningTime="2026-07-08 19:04:29.588023974 +0000 UTC m=+52.846976900" Jul 08 19:04:29.606949 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:29.606907 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-s5dzj" podStartSLOduration=1.178014202 podStartE2EDuration="7.60688888s" totalImagesPullingTime="6.428874678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:04:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:04:29.606304452 +0000 UTC m=+52.865257378" watchObservedRunningTime="2026-07-08 19:04:29.60688888 +0000 UTC m=+52.865841803" Jul 08 19:04:29.628796 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:29.628642 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-fpx7q" podStartSLOduration=2.414492338 podStartE2EDuration="8.628622141s" totalImagesPullingTime="6.214129803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:04:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:04:29.627701279 +0000 UTC m=+52.886654204" watchObservedRunningTime="2026-07-08 19:04:29.628622141 +0000 UTC m=+52.887575066" Jul 08 19:04:29.877777 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:29.877735 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/812854b5-4391-45db-b96d-cddb09c0d324-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-vvggh\" (UID: \"812854b5-4391-45db-b96d-cddb09c0d324\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vvggh" Jul 08 19:04:29.877953 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:29.877899 2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 19:04:29.878010 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:29.877976 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/812854b5-4391-45db-b96d-cddb09c0d324-samples-operator-tls podName:812854b5-4391-45db-b96d-cddb09c0d324 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:37.877956069 +0000 UTC m=+61.136908973 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/812854b5-4391-45db-b96d-cddb09c0d324-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-vvggh" (UID: "812854b5-4391-45db-b96d-cddb09c0d324") : secret "samples-operator-tls" not found Jul 08 19:04:29.978719 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:29.978631 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18b01b38-397e-492c-80c1-61217e74736f-service-ca-bundle\") pod \"router-default-7556c5577f-p4h6b\" (UID: \"18b01b38-397e-492c-80c1-61217e74736f\") " pod="openshift-ingress/router-default-7556c5577f-p4h6b" Jul 08 19:04:29.978863 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:29.978778 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18b01b38-397e-492c-80c1-61217e74736f-service-ca-bundle podName:18b01b38-397e-492c-80c1-61217e74736f nodeName:}" failed. No retries permitted until 2026-07-08 19:04:37.978759329 +0000 UTC m=+61.237712235 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/18b01b38-397e-492c-80c1-61217e74736f-service-ca-bundle") pod "router-default-7556c5577f-p4h6b" (UID: "18b01b38-397e-492c-80c1-61217e74736f") : configmap references non-existent config key: service-ca.crt Jul 08 19:04:29.978863 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:29.978838 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18b01b38-397e-492c-80c1-61217e74736f-metrics-certs\") pod \"router-default-7556c5577f-p4h6b\" (UID: \"18b01b38-397e-492c-80c1-61217e74736f\") " pod="openshift-ingress/router-default-7556c5577f-p4h6b" Jul 08 19:04:29.978989 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:29.978865 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9cebd65f-6a73-46e8-9995-7797c67acb53-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-r7czv\" (UID: \"9cebd65f-6a73-46e8-9995-7797c67acb53\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-r7czv" Jul 08 19:04:29.979040 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:29.978987 2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 19:04:29.979088 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:29.979044 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18b01b38-397e-492c-80c1-61217e74736f-metrics-certs podName:18b01b38-397e-492c-80c1-61217e74736f nodeName:}" failed. No retries permitted until 2026-07-08 19:04:37.979028687 +0000 UTC m=+61.237981594 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18b01b38-397e-492c-80c1-61217e74736f-metrics-certs") pod "router-default-7556c5577f-p4h6b" (UID: "18b01b38-397e-492c-80c1-61217e74736f") : secret "router-metrics-certs-default" not found Jul 08 19:04:29.979088 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:29.978991 2558 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 19:04:29.979195 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:29.979088 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cebd65f-6a73-46e8-9995-7797c67acb53-cluster-monitoring-operator-tls podName:9cebd65f-6a73-46e8-9995-7797c67acb53 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:37.979078566 +0000 UTC m=+61.238031472 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9cebd65f-6a73-46e8-9995-7797c67acb53-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-r7czv" (UID: "9cebd65f-6a73-46e8-9995-7797c67acb53") : secret "cluster-monitoring-operator-tls" not found Jul 08 19:04:30.281763 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:30.281671 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a06e8e9a-6466-46ce-acaa-abdda774607d-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-pr89r\" (UID: \"a06e8e9a-6466-46ce-acaa-abdda774607d\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-pr89r" Jul 08 19:04:30.281919 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:30.281839 2558 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 19:04:30.281919 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:30.281917 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a06e8e9a-6466-46ce-acaa-abdda774607d-networking-console-plugin-cert podName:a06e8e9a-6466-46ce-acaa-abdda774607d nodeName:}" failed. No retries permitted until 2026-07-08 19:04:34.281896781 +0000 UTC m=+57.540849687 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a06e8e9a-6466-46ce-acaa-abdda774607d-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-pr89r" (UID: "a06e8e9a-6466-46ce-acaa-abdda774607d") : secret "networking-console-plugin-cert" not found Jul 08 19:04:30.409083 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:30.409040 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:30.511268 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:30.511226 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:30.530810 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:30.530785 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-54hnr_1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a/console-operator/1.log" Jul 08 19:04:30.531283 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:30.531261 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-54hnr_1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a/console-operator/0.log" Jul 08 19:04:30.531358 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:30.531303 2558 generic.go:356] "Generic (PLEG): container finished" podID="1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a" containerID="558aa3b084fae14496705e317e3caf8ba67c6f1147b6a0340cda6109a2b1b717" exitCode=255 Jul 08 19:04:30.531358 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:30.531333 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-54hnr" event={"ID":"1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a","Type":"ContainerDied","Data":"558aa3b084fae14496705e317e3caf8ba67c6f1147b6a0340cda6109a2b1b717"} Jul 08 19:04:30.531459 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:30.531411 2558 scope.go:117] "RemoveContainer" containerID="73fb4178a72994765c40c8c17e84de7404bb47bd3d7581adb8489b2700290c1f" Jul 08 19:04:30.533362 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:30.532528 2558 scope.go:117] "RemoveContainer" containerID="558aa3b084fae14496705e317e3caf8ba67c6f1147b6a0340cda6109a2b1b717" Jul 08 19:04:30.533362 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:30.532717 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-54hnr_openshift-console-operator(1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a)\"" pod="openshift-console-operator/console-operator-68b757fb67-54hnr" podUID="1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a" Jul 08 19:04:30.629483 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:30.629435 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:30.708567 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:30.708536 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xd9ln"] Jul 08 19:04:30.727058 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:30.727024 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:30.732096 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:30.732069 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xd9ln"] Jul 08 19:04:30.732229 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:30.732188 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xd9ln" Jul 08 19:04:30.734485 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:30.734461 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 08 19:04:30.734801 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:30.734784 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6ts4z\"" Jul 08 19:04:30.734946 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:30.734926 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 08 19:04:30.788498 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:30.788393 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fd304abd-e907-409f-9faf-94c4b78be9d1-data-volume\") pod \"insights-runtime-extractor-xd9ln\" (UID: \"fd304abd-e907-409f-9faf-94c4b78be9d1\") " pod="openshift-insights/insights-runtime-extractor-xd9ln" Jul 08 19:04:30.788666 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:30.788517 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fd304abd-e907-409f-9faf-94c4b78be9d1-crio-socket\") pod \"insights-runtime-extractor-xd9ln\" (UID: \"fd304abd-e907-409f-9faf-94c4b78be9d1\") " pod="openshift-insights/insights-runtime-extractor-xd9ln" Jul 08 19:04:30.788666 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:30.788554 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd304abd-e907-409f-9faf-94c4b78be9d1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xd9ln\" (UID: \"fd304abd-e907-409f-9faf-94c4b78be9d1\") " pod="openshift-insights/insights-runtime-extractor-xd9ln" Jul 08 19:04:30.788666 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:30.788610 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd304abd-e907-409f-9faf-94c4b78be9d1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xd9ln\" (UID: \"fd304abd-e907-409f-9faf-94c4b78be9d1\") " pod="openshift-insights/insights-runtime-extractor-xd9ln" Jul 08 19:04:30.788835 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:30.788723 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v5gc\" (UniqueName: \"kubernetes.io/projected/fd304abd-e907-409f-9faf-94c4b78be9d1-kube-api-access-2v5gc\") pod \"insights-runtime-extractor-xd9ln\" (UID: \"fd304abd-e907-409f-9faf-94c4b78be9d1\") " pod="openshift-insights/insights-runtime-extractor-xd9ln" Jul 08 19:04:30.835144 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:30.835107 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:30.889674 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:30.889642 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fd304abd-e907-409f-9faf-94c4b78be9d1-crio-socket\") pod \"insights-runtime-extractor-xd9ln\" (UID: \"fd304abd-e907-409f-9faf-94c4b78be9d1\") " pod="openshift-insights/insights-runtime-extractor-xd9ln" Jul 08 19:04:30.889791 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:30.889681 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd304abd-e907-409f-9faf-94c4b78be9d1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xd9ln\" (UID: \"fd304abd-e907-409f-9faf-94c4b78be9d1\") " pod="openshift-insights/insights-runtime-extractor-xd9ln" Jul 08 19:04:30.889791 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:30.889720 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd304abd-e907-409f-9faf-94c4b78be9d1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xd9ln\" (UID: \"fd304abd-e907-409f-9faf-94c4b78be9d1\") " pod="openshift-insights/insights-runtime-extractor-xd9ln" Jul 08 19:04:30.889791 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:30.889745 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-registry-tls\") pod \"image-registry-7c4688bdd8-xlx9p\" (UID: \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\") " pod="openshift-image-registry/image-registry-7c4688bdd8-xlx9p" Jul 08 19:04:30.889791 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:30.889769 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v5gc\" (UniqueName: \"kubernetes.io/projected/fd304abd-e907-409f-9faf-94c4b78be9d1-kube-api-access-2v5gc\") pod \"insights-runtime-extractor-xd9ln\" (UID: \"fd304abd-e907-409f-9faf-94c4b78be9d1\") " pod="openshift-insights/insights-runtime-extractor-xd9ln" Jul 08 19:04:30.889941 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:30.889842 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 19:04:30.889941 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:30.889864 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fd304abd-e907-409f-9faf-94c4b78be9d1-crio-socket\") pod \"insights-runtime-extractor-xd9ln\" (UID: \"fd304abd-e907-409f-9faf-94c4b78be9d1\") " pod="openshift-insights/insights-runtime-extractor-xd9ln" Jul 08 19:04:30.889941 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:30.889885 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 19:04:30.889941 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:30.889922 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fd304abd-e907-409f-9faf-94c4b78be9d1-kube-rbac-proxy-cm podName:fd304abd-e907-409f-9faf-94c4b78be9d1 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:31.389902433 +0000 UTC m=+54.648855349 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fd304abd-e907-409f-9faf-94c4b78be9d1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xd9ln" (UID: "fd304abd-e907-409f-9faf-94c4b78be9d1") : configmap "kube-rbac-proxy" not found Jul 08 19:04:30.890097 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:30.889957 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fd304abd-e907-409f-9faf-94c4b78be9d1-data-volume\") pod \"insights-runtime-extractor-xd9ln\" (UID: \"fd304abd-e907-409f-9faf-94c4b78be9d1\") " pod="openshift-insights/insights-runtime-extractor-xd9ln" Jul 08 19:04:30.890097 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:30.889968 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd304abd-e907-409f-9faf-94c4b78be9d1-insights-runtime-extractor-tls podName:fd304abd-e907-409f-9faf-94c4b78be9d1 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:31.389951759 +0000 UTC m=+54.648904669 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fd304abd-e907-409f-9faf-94c4b78be9d1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xd9ln" (UID: "fd304abd-e907-409f-9faf-94c4b78be9d1") : secret "insights-runtime-extractor-tls" not found Jul 08 19:04:30.890097 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:30.889886 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 19:04:30.890097 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:30.889997 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c4688bdd8-xlx9p: secret "image-registry-tls" not found Jul 08 19:04:30.890097 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:30.890040 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-registry-tls podName:fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:46.890027503 +0000 UTC m=+70.148980421 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-registry-tls") pod "image-registry-7c4688bdd8-xlx9p" (UID: "fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7") : secret "image-registry-tls" not found Jul 08 19:04:30.890307 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:30.890206 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fd304abd-e907-409f-9faf-94c4b78be9d1-data-volume\") pod \"insights-runtime-extractor-xd9ln\" (UID: \"fd304abd-e907-409f-9faf-94c4b78be9d1\") " pod="openshift-insights/insights-runtime-extractor-xd9ln" Jul 08 19:04:30.900694 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:30.900658 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:30.906555 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:30.906525 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v5gc\" (UniqueName: \"kubernetes.io/projected/fd304abd-e907-409f-9faf-94c4b78be9d1-kube-api-access-2v5gc\") pod \"insights-runtime-extractor-xd9ln\" (UID: \"fd304abd-e907-409f-9faf-94c4b78be9d1\") " pod="openshift-insights/insights-runtime-extractor-xd9ln" Jul 08 19:04:30.991015 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:30.990985 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eae871c8-0143-47c6-853c-2413d8f6fa70-metrics-tls\") pod \"dns-default-ndtk5\" (UID: \"eae871c8-0143-47c6-853c-2413d8f6fa70\") " pod="openshift-dns/dns-default-ndtk5" Jul 08 19:04:30.991178 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:30.991157 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 19:04:30.991276 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:30.991264 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eae871c8-0143-47c6-853c-2413d8f6fa70-metrics-tls podName:eae871c8-0143-47c6-853c-2413d8f6fa70 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:46.991248084 +0000 UTC m=+70.250200987 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eae871c8-0143-47c6-853c-2413d8f6fa70-metrics-tls") pod "dns-default-ndtk5" (UID: "eae871c8-0143-47c6-853c-2413d8f6fa70") : secret "dns-default-metrics-tls" not found Jul 08 19:04:31.009461 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:31.009433 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:31.100342 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:31.100266 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:31.325659 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:31.325621 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:31.396090 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:31.396062 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd304abd-e907-409f-9faf-94c4b78be9d1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xd9ln\" (UID: \"fd304abd-e907-409f-9faf-94c4b78be9d1\") " pod="openshift-insights/insights-runtime-extractor-xd9ln" Jul 08 19:04:31.396243 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:31.396094 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd304abd-e907-409f-9faf-94c4b78be9d1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xd9ln\" (UID: \"fd304abd-e907-409f-9faf-94c4b78be9d1\") " pod="openshift-insights/insights-runtime-extractor-xd9ln" Jul 08 19:04:31.396243 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:31.396213 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 19:04:31.396314 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:31.396275 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd304abd-e907-409f-9faf-94c4b78be9d1-insights-runtime-extractor-tls podName:fd304abd-e907-409f-9faf-94c4b78be9d1 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:32.396260099 +0000 UTC m=+55.655213028 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fd304abd-e907-409f-9faf-94c4b78be9d1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xd9ln" (UID: "fd304abd-e907-409f-9faf-94c4b78be9d1") : secret "insights-runtime-extractor-tls" not found Jul 08 19:04:31.396314 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:31.396300 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 19:04:31.396383 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:31.396350 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fd304abd-e907-409f-9faf-94c4b78be9d1-kube-rbac-proxy-cm podName:fd304abd-e907-409f-9faf-94c4b78be9d1 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:32.396335576 +0000 UTC m=+55.655288495 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fd304abd-e907-409f-9faf-94c4b78be9d1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xd9ln" (UID: "fd304abd-e907-409f-9faf-94c4b78be9d1") : configmap "kube-rbac-proxy" not found Jul 08 19:04:31.424669 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:31.424638 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:31.536554 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:31.536530 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-54hnr_1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a/console-operator/1.log" Jul 08 19:04:31.536870 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:31.536850 2558 scope.go:117] "RemoveContainer" containerID="558aa3b084fae14496705e317e3caf8ba67c6f1147b6a0340cda6109a2b1b717" Jul 08 19:04:31.537021 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:31.537002 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-54hnr_openshift-console-operator(1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a)\"" pod="openshift-console-operator/console-operator-68b757fb67-54hnr" podUID="1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a" Jul 08 19:04:32.100042 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:32.100013 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-44598_74f5d0c6-337e-407c-bdef-75c0dcd3c47d/dns-node-resolver/0.log" Jul 08 19:04:32.238962 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:32.238932 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-54hnr" Jul 08 19:04:32.239142 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:32.238995 2558 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-54hnr" Jul 08 19:04:32.296187 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:32.296147 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:32.403958 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:32.403921 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd304abd-e907-409f-9faf-94c4b78be9d1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xd9ln\" (UID: \"fd304abd-e907-409f-9faf-94c4b78be9d1\") " pod="openshift-insights/insights-runtime-extractor-xd9ln" Jul 08 19:04:32.403958 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:32.403960 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd304abd-e907-409f-9faf-94c4b78be9d1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xd9ln\" (UID: \"fd304abd-e907-409f-9faf-94c4b78be9d1\") " pod="openshift-insights/insights-runtime-extractor-xd9ln" Jul 08 19:04:32.404322 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:32.404050 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 19:04:32.404322 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:32.404094 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 19:04:32.404322 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:32.404117 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fd304abd-e907-409f-9faf-94c4b78be9d1-kube-rbac-proxy-cm podName:fd304abd-e907-409f-9faf-94c4b78be9d1 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:34.404101808 +0000 UTC m=+57.663054731 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fd304abd-e907-409f-9faf-94c4b78be9d1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xd9ln" (UID: "fd304abd-e907-409f-9faf-94c4b78be9d1") : configmap "kube-rbac-proxy" not found Jul 08 19:04:32.404322 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:32.404155 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd304abd-e907-409f-9faf-94c4b78be9d1-insights-runtime-extractor-tls podName:fd304abd-e907-409f-9faf-94c4b78be9d1 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:34.404139402 +0000 UTC m=+57.663092321 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fd304abd-e907-409f-9faf-94c4b78be9d1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xd9ln" (UID: "fd304abd-e907-409f-9faf-94c4b78be9d1") : secret "insights-runtime-extractor-tls" not found Jul 08 19:04:32.540007 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:32.539983 2558 scope.go:117] "RemoveContainer" containerID="558aa3b084fae14496705e317e3caf8ba67c6f1147b6a0340cda6109a2b1b717" Jul 08 19:04:32.540385 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:32.540128 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-54hnr_openshift-console-operator(1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a)\"" pod="openshift-console-operator/console-operator-68b757fb67-54hnr" podUID="1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a" Jul 08 19:04:32.982524 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:32.982494 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:33.054629 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:33.054573 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:33.298860 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:33.298789 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gktll_a0658e5c-4e22-43d3-ac11-86df5ff39969/node-ca/0.log" Jul 08 19:04:34.102666 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:34.102638 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lng6h_2e731fad-40cc-4fa8-92cf-5465ff9a29dd/serve-healthcheck-canary/0.log" Jul 08 19:04:34.106626 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:34.106598 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:34.321956 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:34.321925 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a06e8e9a-6466-46ce-acaa-abdda774607d-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-pr89r\" (UID: \"a06e8e9a-6466-46ce-acaa-abdda774607d\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-pr89r" Jul 08 19:04:34.322106 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:34.322072 2558 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 19:04:34.322146 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:34.322130 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a06e8e9a-6466-46ce-acaa-abdda774607d-networking-console-plugin-cert podName:a06e8e9a-6466-46ce-acaa-abdda774607d nodeName:}" failed. No retries permitted until 2026-07-08 19:04:42.322115913 +0000 UTC m=+65.581068820 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a06e8e9a-6466-46ce-acaa-abdda774607d-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-pr89r" (UID: "a06e8e9a-6466-46ce-acaa-abdda774607d") : secret "networking-console-plugin-cert" not found Jul 08 19:04:34.423614 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:34.423558 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd304abd-e907-409f-9faf-94c4b78be9d1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xd9ln\" (UID: \"fd304abd-e907-409f-9faf-94c4b78be9d1\") " pod="openshift-insights/insights-runtime-extractor-xd9ln" Jul 08 19:04:34.423896 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:34.423867 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd304abd-e907-409f-9faf-94c4b78be9d1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xd9ln\" (UID: \"fd304abd-e907-409f-9faf-94c4b78be9d1\") " pod="openshift-insights/insights-runtime-extractor-xd9ln" Jul 08 19:04:34.424160 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:34.424142 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 19:04:34.424310 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:34.424299 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fd304abd-e907-409f-9faf-94c4b78be9d1-kube-rbac-proxy-cm podName:fd304abd-e907-409f-9faf-94c4b78be9d1 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:38.424270789 +0000 UTC m=+61.683223708 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fd304abd-e907-409f-9faf-94c4b78be9d1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xd9ln" (UID: "fd304abd-e907-409f-9faf-94c4b78be9d1") : configmap "kube-rbac-proxy" not found Jul 08 19:04:34.424501 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:34.424489 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 19:04:34.424646 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:34.424635 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd304abd-e907-409f-9faf-94c4b78be9d1-insights-runtime-extractor-tls podName:fd304abd-e907-409f-9faf-94c4b78be9d1 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:38.424612883 +0000 UTC m=+61.683565801 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fd304abd-e907-409f-9faf-94c4b78be9d1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xd9ln" (UID: "fd304abd-e907-409f-9faf-94c4b78be9d1") : secret "insights-runtime-extractor-tls" not found Jul 08 19:04:35.487408 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:35.487375 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:35.549506 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:35.549467 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:36.921387 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:36.921351 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:37.951155 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:37.951116 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/812854b5-4391-45db-b96d-cddb09c0d324-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-vvggh\" (UID: \"812854b5-4391-45db-b96d-cddb09c0d324\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vvggh" Jul 08 19:04:37.951696 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:37.951284 2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 19:04:37.951696 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:37.951360 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/812854b5-4391-45db-b96d-cddb09c0d324-samples-operator-tls podName:812854b5-4391-45db-b96d-cddb09c0d324 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:53.951343806 +0000 UTC m=+77.210296709 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/812854b5-4391-45db-b96d-cddb09c0d324-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-vvggh" (UID: "812854b5-4391-45db-b96d-cddb09c0d324") : secret "samples-operator-tls" not found Jul 08 19:04:38.051832 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:38.051793 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18b01b38-397e-492c-80c1-61217e74736f-metrics-certs\") pod \"router-default-7556c5577f-p4h6b\" (UID: \"18b01b38-397e-492c-80c1-61217e74736f\") " pod="openshift-ingress/router-default-7556c5577f-p4h6b" Jul 08 19:04:38.051832 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:38.051834 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9cebd65f-6a73-46e8-9995-7797c67acb53-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-r7czv\" (UID: \"9cebd65f-6a73-46e8-9995-7797c67acb53\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-r7czv" Jul 08 19:04:38.052028 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:38.051901 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18b01b38-397e-492c-80c1-61217e74736f-service-ca-bundle\") pod \"router-default-7556c5577f-p4h6b\" (UID: \"18b01b38-397e-492c-80c1-61217e74736f\") " pod="openshift-ingress/router-default-7556c5577f-p4h6b" Jul 08 19:04:38.052028 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:38.051957 2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 19:04:38.052028 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:38.052001 2558 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 19:04:38.052028 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:38.052014 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18b01b38-397e-492c-80c1-61217e74736f-service-ca-bundle podName:18b01b38-397e-492c-80c1-61217e74736f nodeName:}" failed. No retries permitted until 2026-07-08 19:04:54.052001733 +0000 UTC m=+77.310954635 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/18b01b38-397e-492c-80c1-61217e74736f-service-ca-bundle") pod "router-default-7556c5577f-p4h6b" (UID: "18b01b38-397e-492c-80c1-61217e74736f") : configmap references non-existent config key: service-ca.crt Jul 08 19:04:38.052170 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:38.052043 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cebd65f-6a73-46e8-9995-7797c67acb53-cluster-monitoring-operator-tls podName:9cebd65f-6a73-46e8-9995-7797c67acb53 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:54.052031222 +0000 UTC m=+77.310984125 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9cebd65f-6a73-46e8-9995-7797c67acb53-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-r7czv" (UID: "9cebd65f-6a73-46e8-9995-7797c67acb53") : secret "cluster-monitoring-operator-tls" not found Jul 08 19:04:38.052170 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:38.052065 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18b01b38-397e-492c-80c1-61217e74736f-metrics-certs podName:18b01b38-397e-492c-80c1-61217e74736f nodeName:}" failed. No retries permitted until 2026-07-08 19:04:54.052059027 +0000 UTC m=+77.311011930 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18b01b38-397e-492c-80c1-61217e74736f-metrics-certs") pod "router-default-7556c5577f-p4h6b" (UID: "18b01b38-397e-492c-80c1-61217e74736f") : secret "router-metrics-certs-default" not found Jul 08 19:04:38.455785 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:38.455758 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd304abd-e907-409f-9faf-94c4b78be9d1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xd9ln\" (UID: \"fd304abd-e907-409f-9faf-94c4b78be9d1\") " pod="openshift-insights/insights-runtime-extractor-xd9ln" Jul 08 19:04:38.455785 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:38.455789 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd304abd-e907-409f-9faf-94c4b78be9d1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xd9ln\" (UID: \"fd304abd-e907-409f-9faf-94c4b78be9d1\") " pod="openshift-insights/insights-runtime-extractor-xd9ln" Jul 08 19:04:38.456027 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:38.455920 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 19:04:38.456027 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:38.455945 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 19:04:38.456027 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:38.456007 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd304abd-e907-409f-9faf-94c4b78be9d1-insights-runtime-extractor-tls podName:fd304abd-e907-409f-9faf-94c4b78be9d1 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:46.45598364 +0000 UTC m=+69.714936545 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fd304abd-e907-409f-9faf-94c4b78be9d1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xd9ln" (UID: "fd304abd-e907-409f-9faf-94c4b78be9d1") : secret "insights-runtime-extractor-tls" not found Jul 08 19:04:38.456027 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:38.456026 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fd304abd-e907-409f-9faf-94c4b78be9d1-kube-rbac-proxy-cm podName:fd304abd-e907-409f-9faf-94c4b78be9d1 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:46.456016315 +0000 UTC m=+69.714969221 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fd304abd-e907-409f-9faf-94c4b78be9d1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xd9ln" (UID: "fd304abd-e907-409f-9faf-94c4b78be9d1") : configmap "kube-rbac-proxy" not found Jul 08 19:04:39.473810 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:39.473770 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:39.616416 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:39.616383 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:39.723880 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:39.723791 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:39.825365 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:39.825331 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:40.050806 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:40.050714 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:40.163030 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:40.162993 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:40.288522 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:40.288489 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:41.188000 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:41.187960 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:41.449460 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:41.449392 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6gmjt" Jul 08 19:04:41.580212 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:41.580177 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:41.790087 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:41.789996 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:41.994296 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:41.994258 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:42.189671 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:42.189638 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:42.388737 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:42.388696 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a06e8e9a-6466-46ce-acaa-abdda774607d-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-pr89r\" (UID: \"a06e8e9a-6466-46ce-acaa-abdda774607d\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-pr89r" Jul 08 19:04:42.388912 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:42.388871 2558 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 19:04:42.388959 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:42.388942 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a06e8e9a-6466-46ce-acaa-abdda774607d-networking-console-plugin-cert podName:a06e8e9a-6466-46ce-acaa-abdda774607d nodeName:}" failed. No retries permitted until 2026-07-08 19:04:58.388926608 +0000 UTC m=+81.647879511 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a06e8e9a-6466-46ce-acaa-abdda774607d-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-pr89r" (UID: "a06e8e9a-6466-46ce-acaa-abdda774607d") : secret "networking-console-plugin-cert" not found Jul 08 19:04:42.655952 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:42.655921 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:42.757319 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:42.757282 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:42.904099 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:42.904063 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:43.171662 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:43.171627 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:43.196825 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:43.196800 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/640044e9-7c83-4f5a-9c9d-2e4183284806-original-pull-secret\") pod \"global-pull-secret-syncer-mlw2c\" (UID: \"640044e9-7c83-4f5a-9c9d-2e4183284806\") " pod="kube-system/global-pull-secret-syncer-mlw2c" Jul 08 19:04:43.197194 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:43.196857 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99034b7e-72e2-4a28-8711-233630b7e9ea-metrics-certs\") pod \"network-metrics-daemon-5rmts\" (UID: \"99034b7e-72e2-4a28-8711-233630b7e9ea\") " pod="openshift-multus/network-metrics-daemon-5rmts" Jul 08 19:04:43.200643 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:43.200626 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 08 19:04:43.201180 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:43.201166 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 08 19:04:43.207370 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:43.207352 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 08 19:04:43.207434 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:43.207423 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99034b7e-72e2-4a28-8711-233630b7e9ea-metrics-certs podName:99034b7e-72e2-4a28-8711-233630b7e9ea nodeName:}" failed. No retries permitted until 2026-07-08 19:05:47.207402902 +0000 UTC m=+130.466355808 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99034b7e-72e2-4a28-8711-233630b7e9ea-metrics-certs") pod "network-metrics-daemon-5rmts" (UID: "99034b7e-72e2-4a28-8711-233630b7e9ea") : secret "metrics-daemon-secret" not found Jul 08 19:04:43.209482 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:43.209466 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/640044e9-7c83-4f5a-9c9d-2e4183284806-original-pull-secret\") pod \"global-pull-secret-syncer-mlw2c\" (UID: \"640044e9-7c83-4f5a-9c9d-2e4183284806\") " pod="kube-system/global-pull-secret-syncer-mlw2c" Jul 08 19:04:43.298035 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:43.298007 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lszs7\" (UniqueName: \"kubernetes.io/projected/6fcd111d-1192-4c3e-90e7-2294cdec33ee-kube-api-access-lszs7\") pod \"network-check-target-26v6h\" (UID: \"6fcd111d-1192-4c3e-90e7-2294cdec33ee\") " pod="openshift-network-diagnostics/network-check-target-26v6h" Jul 08 19:04:43.300415 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:43.300390 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lszs7\" (UniqueName: \"kubernetes.io/projected/6fcd111d-1192-4c3e-90e7-2294cdec33ee-kube-api-access-lszs7\") pod \"network-check-target-26v6h\" (UID: \"6fcd111d-1192-4c3e-90e7-2294cdec33ee\") " pod="openshift-network-diagnostics/network-check-target-26v6h" Jul 08 19:04:43.384023 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:43.383990 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:43.489462 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:43.489372 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gzqsf\"" Jul 08 19:04:43.496647 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:43.496622 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26v6h" Jul 08 19:04:43.496727 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:43.496623 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlw2c" Jul 08 19:04:43.596183 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:43.596034 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:43.634183 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:43.634162 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-26v6h"] Jul 08 19:04:43.635477 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:04:43.635452 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fcd111d_1192_4c3e_90e7_2294cdec33ee.slice/crio-34227d3ab218d10f96b34b0967df452523da4809ab769be4e1041a23c901aa1b WatchSource:0}: Error finding container 34227d3ab218d10f96b34b0967df452523da4809ab769be4e1041a23c901aa1b: Status 404 returned error can't find the container with id 34227d3ab218d10f96b34b0967df452523da4809ab769be4e1041a23c901aa1b Jul 08 19:04:43.649173 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:43.649152 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mlw2c"] Jul 08 19:04:43.652082 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:04:43.652061 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod640044e9_7c83_4f5a_9c9d_2e4183284806.slice/crio-2e0296e96d6e7b81e674a8c9c32477c1b125991087be2baf1ce822c7a151167b WatchSource:0}: Error finding container 2e0296e96d6e7b81e674a8c9c32477c1b125991087be2baf1ce822c7a151167b: Status 404 returned error can't find the container with id 2e0296e96d6e7b81e674a8c9c32477c1b125991087be2baf1ce822c7a151167b Jul 08 19:04:43.802801 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:43.802717 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:43.913149 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:43.913112 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:44.003283 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:44.003248 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:44.357226 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:44.357127 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:44.422088 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:44.422054 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:44.576790 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:44.576753 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-26v6h" event={"ID":"6fcd111d-1192-4c3e-90e7-2294cdec33ee","Type":"ContainerStarted","Data":"e99f7a552edf0fb17aa04cf162b8f8c2cc6719d879b75e985681d046ef01ac9b"} Jul 08 19:04:44.576961 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:44.576799 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-26v6h" event={"ID":"6fcd111d-1192-4c3e-90e7-2294cdec33ee","Type":"ContainerStarted","Data":"34227d3ab218d10f96b34b0967df452523da4809ab769be4e1041a23c901aa1b"} Jul 08 19:04:44.576961 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:44.576843 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-26v6h" Jul 08 19:04:44.577988 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:44.577966 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mlw2c" event={"ID":"640044e9-7c83-4f5a-9c9d-2e4183284806","Type":"ContainerStarted","Data":"2e0296e96d6e7b81e674a8c9c32477c1b125991087be2baf1ce822c7a151167b"} Jul 08 19:04:44.594908 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:44.594865 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-26v6h" podStartSLOduration=67.594852396 podStartE2EDuration="1m7.594852396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:03:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:04:44.594042963 +0000 UTC m=+67.852995899" watchObservedRunningTime="2026-07-08 19:04:44.594852396 +0000 UTC m=+67.853805318" Jul 08 19:04:45.180993 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:45.180904 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:45.367831 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:45.367794 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:45.466986 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:45.466906 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:45.581957 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:45.581923 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:46.014285 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:46.014250 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:46.116866 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:46.116818 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:46.276224 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:46.276143 2558 scope.go:117] "RemoveContainer" containerID="558aa3b084fae14496705e317e3caf8ba67c6f1147b6a0340cda6109a2b1b717" Jul 08 19:04:46.527055 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:46.526973 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd304abd-e907-409f-9faf-94c4b78be9d1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xd9ln\" (UID: \"fd304abd-e907-409f-9faf-94c4b78be9d1\") " pod="openshift-insights/insights-runtime-extractor-xd9ln" Jul 08 19:04:46.527055 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:46.527026 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd304abd-e907-409f-9faf-94c4b78be9d1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xd9ln\" (UID: \"fd304abd-e907-409f-9faf-94c4b78be9d1\") " pod="openshift-insights/insights-runtime-extractor-xd9ln" Jul 08 19:04:46.527550 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:46.527085 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 19:04:46.527550 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:46.527164 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd304abd-e907-409f-9faf-94c4b78be9d1-insights-runtime-extractor-tls podName:fd304abd-e907-409f-9faf-94c4b78be9d1 nodeName:}" failed. No retries permitted until 2026-07-08 19:05:02.52714603 +0000 UTC m=+85.786098933 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fd304abd-e907-409f-9faf-94c4b78be9d1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xd9ln" (UID: "fd304abd-e907-409f-9faf-94c4b78be9d1") : secret "insights-runtime-extractor-tls" not found Jul 08 19:04:46.527650 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:46.527633 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd304abd-e907-409f-9faf-94c4b78be9d1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xd9ln\" (UID: \"fd304abd-e907-409f-9faf-94c4b78be9d1\") " pod="openshift-insights/insights-runtime-extractor-xd9ln" Jul 08 19:04:46.587740 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:46.587709 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-54hnr_1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a/console-operator/2.log" Jul 08 19:04:46.588228 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:46.588196 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-54hnr_1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a/console-operator/1.log" Jul 08 19:04:46.588352 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:46.588240 2558 generic.go:356] "Generic (PLEG): container finished" podID="1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a" containerID="ad3aeb016e8b6852da165019f9ba33a261a8599f252b9b194a814f016aa04545" exitCode=255 Jul 08 19:04:46.588352 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:46.588308 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-54hnr" event={"ID":"1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a","Type":"ContainerDied","Data":"ad3aeb016e8b6852da165019f9ba33a261a8599f252b9b194a814f016aa04545"} Jul 08 19:04:46.588468 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:46.588364 2558 scope.go:117] "RemoveContainer" containerID="558aa3b084fae14496705e317e3caf8ba67c6f1147b6a0340cda6109a2b1b717" Jul 08 19:04:46.588704 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:46.588680 2558 scope.go:117] "RemoveContainer" containerID="ad3aeb016e8b6852da165019f9ba33a261a8599f252b9b194a814f016aa04545" Jul 08 19:04:46.588921 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:46.588885 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-68b757fb67-54hnr_openshift-console-operator(1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a)\"" pod="openshift-console-operator/console-operator-68b757fb67-54hnr" podUID="1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a" Jul 08 19:04:46.778073 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:46.777981 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:46.930589 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:46.930555 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-registry-tls\") pod \"image-registry-7c4688bdd8-xlx9p\" (UID: \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\") " pod="openshift-image-registry/image-registry-7c4688bdd8-xlx9p" Jul 08 19:04:46.933181 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:46.933155 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-registry-tls\") pod \"image-registry-7c4688bdd8-xlx9p\" (UID: \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\") " pod="openshift-image-registry/image-registry-7c4688bdd8-xlx9p" Jul 08 19:04:47.031810 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:47.031720 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eae871c8-0143-47c6-853c-2413d8f6fa70-metrics-tls\") pod \"dns-default-ndtk5\" (UID: \"eae871c8-0143-47c6-853c-2413d8f6fa70\") " pod="openshift-dns/dns-default-ndtk5" Jul 08 19:04:47.034350 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:47.034321 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eae871c8-0143-47c6-853c-2413d8f6fa70-metrics-tls\") pod \"dns-default-ndtk5\" (UID: \"eae871c8-0143-47c6-853c-2413d8f6fa70\") " pod="openshift-dns/dns-default-ndtk5" Jul 08 19:04:47.178308 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:47.178269 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:47.195710 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:47.195681 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zm6q7\"" Jul 08 19:04:47.204087 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:47.204065 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c4688bdd8-xlx9p" Jul 08 19:04:47.265721 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:47.265692 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xm8m6\"" Jul 08 19:04:47.273340 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:47.273317 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ndtk5" Jul 08 19:04:47.383357 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:47.383288 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:47.580431 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:47.580392 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:47.594396 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:47.594371 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-54hnr_1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a/console-operator/2.log" Jul 08 19:04:47.981334 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:47.981216 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:48.328086 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:48.328057 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c4688bdd8-xlx9p"] Jul 08 19:04:48.331012 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:04:48.330973 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbb20ea6_abfe_4671_8dc7_4a2e15cef3c7.slice/crio-2bd365595046a79bfe202ef29774895802323d407c120a070675fd50a71dbd2f WatchSource:0}: Error finding container 2bd365595046a79bfe202ef29774895802323d407c120a070675fd50a71dbd2f: Status 404 returned error can't find the container with id 2bd365595046a79bfe202ef29774895802323d407c120a070675fd50a71dbd2f Jul 08 19:04:48.331676 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:48.331647 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ndtk5"] Jul 08 19:04:48.334245 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:04:48.334223 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeae871c8_0143_47c6_853c_2413d8f6fa70.slice/crio-df52717bf70fee0977d236ce1772bd1a2988e910d1af8d3e43354bb630db06d3 WatchSource:0}: Error finding container df52717bf70fee0977d236ce1772bd1a2988e910d1af8d3e43354bb630db06d3: Status 404 returned error can't find the container with id df52717bf70fee0977d236ce1772bd1a2988e910d1af8d3e43354bb630db06d3 Jul 08 19:04:48.578431 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:48.578342 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:48.601850 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:48.601805 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mlw2c" event={"ID":"640044e9-7c83-4f5a-9c9d-2e4183284806","Type":"ContainerStarted","Data":"92165843a5cf734a77b094b2a7f300795b55184d6826b4a0ca69923e84d5965c"} Jul 08 19:04:48.603342 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:48.603314 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c4688bdd8-xlx9p" event={"ID":"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7","Type":"ContainerStarted","Data":"23030c088d7a62df04cd0775a36580feafab4fc9b55ca63d6cc6c2b173dc1082"} Jul 08 19:04:48.603484 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:48.603362 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c4688bdd8-xlx9p" event={"ID":"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7","Type":"ContainerStarted","Data":"2bd365595046a79bfe202ef29774895802323d407c120a070675fd50a71dbd2f"} Jul 08 19:04:48.603484 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:48.603393 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7c4688bdd8-xlx9p" Jul 08 19:04:48.604457 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:48.604433 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ndtk5" event={"ID":"eae871c8-0143-47c6-853c-2413d8f6fa70","Type":"ContainerStarted","Data":"df52717bf70fee0977d236ce1772bd1a2988e910d1af8d3e43354bb630db06d3"} Jul 08 19:04:48.621328 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:48.621281 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mlw2c" podStartSLOduration=66.074160031 podStartE2EDuration="1m10.6212619s" totalImagesPullingTime="4.547101869s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:03:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:04:48.619769382 +0000 UTC m=+71.878722308" watchObservedRunningTime="2026-07-08 19:04:48.6212619 +0000 UTC m=+71.880214824" Jul 08 19:04:48.643135 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:48.643092 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7c4688bdd8-xlx9p" podStartSLOduration=70.643077182 podStartE2EDuration="1m10.643077182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:03:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:04:48.641664872 +0000 UTC m=+71.900617795" watchObservedRunningTime="2026-07-08 19:04:48.643077182 +0000 UTC m=+71.902030107" Jul 08 19:04:48.982571 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:48.982530 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:49.176137 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:49.176079 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:49.587652 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:49.587615 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:49.981749 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:49.981715 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:50.194566 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:50.194541 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:50.613313 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:50.613223 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ndtk5" event={"ID":"eae871c8-0143-47c6-853c-2413d8f6fa70","Type":"ContainerStarted","Data":"b5ebf8575fa28facbd44ac4992d39df490ccafef222c3bf6bb53b45309dc0f39"} Jul 08 19:04:50.613313 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:50.613308 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ndtk5" event={"ID":"eae871c8-0143-47c6-853c-2413d8f6fa70","Type":"ContainerStarted","Data":"21c3a1513ca80adee42d9eb2adfd59b78624988f9c38ee00ad3964f7a2906d8b"} Jul 08 19:04:50.613487 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:50.613343 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ndtk5" Jul 08 19:04:50.631347 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:50.631297 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ndtk5" podStartSLOduration=34.21424634 podStartE2EDuration="35.631279903s" totalImagesPullingTime="1.417033563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:04:50.630732126 +0000 UTC m=+73.889685060" watchObservedRunningTime="2026-07-08 19:04:50.631279903 +0000 UTC m=+73.890232827" Jul 08 19:04:52.239627 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:52.239555 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-54hnr" Jul 08 19:04:52.239993 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:52.239647 2558 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-54hnr" Jul 08 19:04:52.239993 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:52.239966 2558 scope.go:117] "RemoveContainer" containerID="ad3aeb016e8b6852da165019f9ba33a261a8599f252b9b194a814f016aa04545" Jul 08 19:04:52.240138 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:04:52.240118 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-68b757fb67-54hnr_openshift-console-operator(1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a)\"" pod="openshift-console-operator/console-operator-68b757fb67-54hnr" podUID="1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a" Jul 08 19:04:53.998840 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:53.998793 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/812854b5-4391-45db-b96d-cddb09c0d324-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-vvggh\" (UID: \"812854b5-4391-45db-b96d-cddb09c0d324\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vvggh" Jul 08 19:04:54.001427 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:54.001393 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/812854b5-4391-45db-b96d-cddb09c0d324-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-vvggh\" (UID: \"812854b5-4391-45db-b96d-cddb09c0d324\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vvggh" Jul 08 19:04:54.099333 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:54.099299 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18b01b38-397e-492c-80c1-61217e74736f-metrics-certs\") pod \"router-default-7556c5577f-p4h6b\" (UID: \"18b01b38-397e-492c-80c1-61217e74736f\") " pod="openshift-ingress/router-default-7556c5577f-p4h6b" Jul 08 19:04:54.099333 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:54.099338 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9cebd65f-6a73-46e8-9995-7797c67acb53-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-r7czv\" (UID: \"9cebd65f-6a73-46e8-9995-7797c67acb53\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-r7czv" Jul 08 19:04:54.099528 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:54.099407 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18b01b38-397e-492c-80c1-61217e74736f-service-ca-bundle\") pod \"router-default-7556c5577f-p4h6b\" (UID: \"18b01b38-397e-492c-80c1-61217e74736f\") " pod="openshift-ingress/router-default-7556c5577f-p4h6b" Jul 08 19:04:54.100091 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:54.100061 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18b01b38-397e-492c-80c1-61217e74736f-service-ca-bundle\") pod \"router-default-7556c5577f-p4h6b\" (UID: \"18b01b38-397e-492c-80c1-61217e74736f\") " pod="openshift-ingress/router-default-7556c5577f-p4h6b" Jul 08 19:04:54.101718 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:54.101696 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9cebd65f-6a73-46e8-9995-7797c67acb53-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-r7czv\" (UID: \"9cebd65f-6a73-46e8-9995-7797c67acb53\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-r7czv" Jul 08 19:04:54.101718 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:54.101711 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18b01b38-397e-492c-80c1-61217e74736f-metrics-certs\") pod \"router-default-7556c5577f-p4h6b\" (UID: \"18b01b38-397e-492c-80c1-61217e74736f\") " pod="openshift-ingress/router-default-7556c5577f-p4h6b" Jul 08 19:04:54.270148 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:54.270070 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-nnzht\"" Jul 08 19:04:54.277792 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:54.277771 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vvggh" Jul 08 19:04:54.308343 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:54.308314 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-gxtbn\"" Jul 08 19:04:54.316431 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:54.316409 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7556c5577f-p4h6b" Jul 08 19:04:54.331359 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:54.331332 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-lf8n5\"" Jul 08 19:04:54.338999 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:54.338970 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-r7czv" Jul 08 19:04:54.419847 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:54.417236 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vvggh"] Jul 08 19:04:54.481197 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:54.481172 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7556c5577f-p4h6b"] Jul 08 19:04:54.483161 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:04:54.483136 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18b01b38_397e_492c_80c1_61217e74736f.slice/crio-4ba965262eae7a3a1b6d041b61ee35a6bde3f68e447f8dd98ff06f0ef63ed0e4 WatchSource:0}: Error finding container 4ba965262eae7a3a1b6d041b61ee35a6bde3f68e447f8dd98ff06f0ef63ed0e4: Status 404 returned error can't find the container with id 4ba965262eae7a3a1b6d041b61ee35a6bde3f68e447f8dd98ff06f0ef63ed0e4 Jul 08 19:04:54.518519 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:54.518495 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-r7czv"] Jul 08 19:04:54.525628 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:04:54.525599 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cebd65f_6a73_46e8_9995_7797c67acb53.slice/crio-d38bba0aa31814f2e1d9ecf1e6c4b96a55113fc50e3bfc2d51e6b7f32bead9f8 WatchSource:0}: Error finding container d38bba0aa31814f2e1d9ecf1e6c4b96a55113fc50e3bfc2d51e6b7f32bead9f8: Status 404 returned error can't find the container with id d38bba0aa31814f2e1d9ecf1e6c4b96a55113fc50e3bfc2d51e6b7f32bead9f8 Jul 08 19:04:54.629023 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:54.628981 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-r7czv" event={"ID":"9cebd65f-6a73-46e8-9995-7797c67acb53","Type":"ContainerStarted","Data":"d38bba0aa31814f2e1d9ecf1e6c4b96a55113fc50e3bfc2d51e6b7f32bead9f8"} Jul 08 19:04:54.630528 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:54.630502 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7556c5577f-p4h6b" event={"ID":"18b01b38-397e-492c-80c1-61217e74736f","Type":"ContainerStarted","Data":"6a2b60ce8f7e76000bbab54fab899aed96299b8ac82e2eb97e09c76dc7156931"} Jul 08 19:04:54.630697 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:54.630542 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7556c5577f-p4h6b" event={"ID":"18b01b38-397e-492c-80c1-61217e74736f","Type":"ContainerStarted","Data":"4ba965262eae7a3a1b6d041b61ee35a6bde3f68e447f8dd98ff06f0ef63ed0e4"} Jul 08 19:04:54.631642 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:54.631616 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vvggh" event={"ID":"812854b5-4391-45db-b96d-cddb09c0d324","Type":"ContainerStarted","Data":"2b91d24d87bd9075b760f740a679fe954b31e5cca45282aef2185d060844d7c8"} Jul 08 19:04:54.681711 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:54.681663 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7556c5577f-p4h6b" podStartSLOduration=32.681647968 podStartE2EDuration="32.681647968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:04:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:04:54.679997441 +0000 UTC m=+77.938950365" watchObservedRunningTime="2026-07-08 19:04:54.681647968 +0000 UTC m=+77.940600892" Jul 08 19:04:54.737207 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:54.737167 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:54.831691 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:54.831608 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:54.927310 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:54.927279 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:55.030381 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:55.030353 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:55.132447 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:55.132406 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:55.317060 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:55.317025 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7556c5577f-p4h6b" Jul 08 19:04:55.319591 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:55.319552 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7556c5577f-p4h6b" Jul 08 19:04:55.636393 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:55.636365 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7556c5577f-p4h6b" Jul 08 19:04:55.637782 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:55.637757 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7556c5577f-p4h6b" Jul 08 19:04:56.977419 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:56.977386 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:57.037526 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:57.037491 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:57.516268 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:57.516230 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:57.589098 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:57.589066 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:58.222297 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:58.222257 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:58.327068 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:58.326986 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:58.333335 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:58.333313 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7ssjd"] Jul 08 19:04:58.349182 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:58.349160 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7ssjd" Jul 08 19:04:58.351848 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:58.351825 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-64wvn\"" Jul 08 19:04:58.352022 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:58.351855 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 08 19:04:58.356136 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:58.356038 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7ssjd"] Jul 08 19:04:58.402897 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:58.402863 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:58.436967 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:58.436933 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3aa66b57-e57d-4411-8e4a-4afa9e33b34b-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-7ssjd\" (UID: \"3aa66b57-e57d-4411-8e4a-4afa9e33b34b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7ssjd" Jul 08 19:04:58.437115 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:58.436994 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a06e8e9a-6466-46ce-acaa-abdda774607d-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-pr89r\" (UID: \"a06e8e9a-6466-46ce-acaa-abdda774607d\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-pr89r" Jul 08 19:04:58.439279 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:58.439253 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a06e8e9a-6466-46ce-acaa-abdda774607d-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-pr89r\" (UID: \"a06e8e9a-6466-46ce-acaa-abdda774607d\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-pr89r" Jul 08 19:04:58.538034 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:58.537996 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3aa66b57-e57d-4411-8e4a-4afa9e33b34b-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-7ssjd\" (UID: \"3aa66b57-e57d-4411-8e4a-4afa9e33b34b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7ssjd" Jul 08 19:04:58.540460 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:58.540429 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3aa66b57-e57d-4411-8e4a-4afa9e33b34b-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-7ssjd\" (UID: \"3aa66b57-e57d-4411-8e4a-4afa9e33b34b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7ssjd" Jul 08 19:04:58.558141 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:58.558088 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:04:58.648753 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:58.648713 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-r7czv" event={"ID":"9cebd65f-6a73-46e8-9995-7797c67acb53","Type":"ContainerStarted","Data":"a537e80f537a51ef9170f8430107006f63b1055a4b037f4676130178bf10615a"} Jul 08 19:04:58.650442 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:58.650417 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vvggh" event={"ID":"812854b5-4391-45db-b96d-cddb09c0d324","Type":"ContainerStarted","Data":"d5d0c3eb37dbb01775fcb922f91d0d18212ed9d9e59898add4b048f553856a66"} Jul 08 19:04:58.650541 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:58.650465 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vvggh" event={"ID":"812854b5-4391-45db-b96d-cddb09c0d324","Type":"ContainerStarted","Data":"33f2596eacf93ebaf028ea1757ada31169f6dbc21126d5e5caebb9522fd750a4"} Jul 08 19:04:58.654001 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:58.653984 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-cktkj\"" Jul 08 19:04:58.659162 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:58.659143 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7ssjd" Jul 08 19:04:58.662262 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:58.662247 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-pr89r" Jul 08 19:04:58.673268 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:58.673229 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-r7czv" podStartSLOduration=33.403291084 podStartE2EDuration="36.673217598s" totalImagesPullingTime="3.269926514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:04:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:04:58.671994155 +0000 UTC m=+81.930947106" watchObservedRunningTime="2026-07-08 19:04:58.673217598 +0000 UTC m=+81.932170570" Jul 08 19:04:58.691510 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:58.691460 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vvggh" podStartSLOduration=33.351340588 podStartE2EDuration="36.691442075s" totalImagesPullingTime="3.340101487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:04:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:04:58.689943895 +0000 UTC m=+81.948896819" watchObservedRunningTime="2026-07-08 19:04:58.691442075 +0000 UTC m=+81.950395198" Jul 08 19:04:58.822091 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:58.822058 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7ssjd"] Jul 08 19:04:58.825703 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:04:58.825672 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa66b57_e57d_4411_8e4a_4afa9e33b34b.slice/crio-7c58eadbbfd14abc03bd3d5f31731c918a6c304affca9c3e4fc81fe37fe3d57b WatchSource:0}: Error finding container 7c58eadbbfd14abc03bd3d5f31731c918a6c304affca9c3e4fc81fe37fe3d57b: Status 404 returned error can't find the container with id 7c58eadbbfd14abc03bd3d5f31731c918a6c304affca9c3e4fc81fe37fe3d57b Jul 08 19:04:58.846437 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:58.846390 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-pr89r"] Jul 08 19:04:58.848991 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:04:58.848968 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda06e8e9a_6466_46ce_acaa_abdda774607d.slice/crio-65c63dbc0bfe5f71a027b86f73d5820e2354dd9112f2051c05e99e1666a9bd10 WatchSource:0}: Error finding container 65c63dbc0bfe5f71a027b86f73d5820e2354dd9112f2051c05e99e1666a9bd10: Status 404 returned error can't find the container with id 65c63dbc0bfe5f71a027b86f73d5820e2354dd9112f2051c05e99e1666a9bd10 Jul 08 19:04:59.655111 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:59.654975 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-pr89r" event={"ID":"a06e8e9a-6466-46ce-acaa-abdda774607d","Type":"ContainerStarted","Data":"65c63dbc0bfe5f71a027b86f73d5820e2354dd9112f2051c05e99e1666a9bd10"} Jul 08 19:04:59.656497 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:04:59.656465 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7ssjd" event={"ID":"3aa66b57-e57d-4411-8e4a-4afa9e33b34b","Type":"ContainerStarted","Data":"7c58eadbbfd14abc03bd3d5f31731c918a6c304affca9c3e4fc81fe37fe3d57b"} Jul 08 19:05:00.535662 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:00.535632 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:00.596721 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:00.596685 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:00.619505 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:00.619475 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ndtk5" Jul 08 19:05:00.663315 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:00.663276 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7ssjd" event={"ID":"3aa66b57-e57d-4411-8e4a-4afa9e33b34b","Type":"ContainerStarted","Data":"0e4e316531f260680f71701a5bcb772f1fa955bb44d1eb5eb3a1fc8397fddc04"} Jul 08 19:05:00.663776 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:00.663589 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7ssjd" Jul 08 19:05:00.669109 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:00.669057 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7ssjd" Jul 08 19:05:00.680126 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:00.680081 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7ssjd" podStartSLOduration=1.073733122 podStartE2EDuration="2.680066573s" totalImagesPullingTime="1.606333451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:04:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:05:00.679821841 +0000 UTC m=+83.938774768" watchObservedRunningTime="2026-07-08 19:05:00.680066573 +0000 UTC m=+83.939019498" Jul 08 19:05:01.358673 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:01.358640 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:01.441597 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:01.441550 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:01.668002 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:01.667960 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-pr89r" event={"ID":"a06e8e9a-6466-46ce-acaa-abdda774607d","Type":"ContainerStarted","Data":"4b2ad9a0358ec47740b8bf128d2eba3265ddef33745d243b61ba00540d7b7368"} Jul 08 19:05:01.686463 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:01.686405 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-pr89r" podStartSLOduration=33.248886323 podStartE2EDuration="35.68638925s" totalImagesPullingTime="2.437502927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:04:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:05:01.685558252 +0000 UTC m=+84.944511178" watchObservedRunningTime="2026-07-08 19:05:01.68638925 +0000 UTC m=+84.945342174" Jul 08 19:05:01.839826 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:01.839787 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:01.909591 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:01.909538 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:01.977277 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:01.977188 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:02.413316 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:02.413253 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:02.514306 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:02.514281 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:02.566689 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:02.566649 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd304abd-e907-409f-9faf-94c4b78be9d1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xd9ln\" (UID: \"fd304abd-e907-409f-9faf-94c4b78be9d1\") " pod="openshift-insights/insights-runtime-extractor-xd9ln" Jul 08 19:05:02.568908 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:02.568891 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd304abd-e907-409f-9faf-94c4b78be9d1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xd9ln\" (UID: \"fd304abd-e907-409f-9faf-94c4b78be9d1\") " pod="openshift-insights/insights-runtime-extractor-xd9ln" Jul 08 19:05:02.846998 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:02.846924 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6ts4z\"" Jul 08 19:05:02.854348 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:02.854331 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xd9ln" Jul 08 19:05:02.981304 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:02.981176 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xd9ln"] Jul 08 19:05:02.983897 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:05:02.983869 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd304abd_e907_409f_9faf_94c4b78be9d1.slice/crio-0c9efe03ff3c7f3c235654198449dcc50618801058d7d819cfe9752689dca4bd WatchSource:0}: Error finding container 0c9efe03ff3c7f3c235654198449dcc50618801058d7d819cfe9752689dca4bd: Status 404 returned error can't find the container with id 0c9efe03ff3c7f3c235654198449dcc50618801058d7d819cfe9752689dca4bd Jul 08 19:05:03.629514 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:03.629389 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:03.678156 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:03.678060 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xd9ln" event={"ID":"fd304abd-e907-409f-9faf-94c4b78be9d1","Type":"ContainerStarted","Data":"0c9efe03ff3c7f3c235654198449dcc50618801058d7d819cfe9752689dca4bd"} Jul 08 19:05:03.730776 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:03.730725 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:07.208619 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:07.208563 2558 patch_prober.go:28] interesting pod/image-registry-7c4688bdd8-xlx9p container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 08 19:05:07.209087 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:07.208645 2558 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7c4688bdd8-xlx9p" podUID="fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 08 19:05:07.277970 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:07.277945 2558 scope.go:117] "RemoveContainer" containerID="ad3aeb016e8b6852da165019f9ba33a261a8599f252b9b194a814f016aa04545" Jul 08 19:05:07.694112 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:07.694080 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-54hnr_1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a/console-operator/2.log" Jul 08 19:05:07.694293 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:07.694188 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-54hnr" event={"ID":"1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a","Type":"ContainerStarted","Data":"911900cf116b5699c9d5a53aece48f9b7ebdb22679d5355ed95c4b0ec682c288"} Jul 08 19:05:07.694515 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:07.694493 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-54hnr" Jul 08 19:05:07.696381 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:07.696354 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xd9ln" event={"ID":"fd304abd-e907-409f-9faf-94c4b78be9d1","Type":"ContainerStarted","Data":"0c16224342a8172b6c2b69e70aaaf5136a8cc2e805ba369b0a4c62924c83252f"} Jul 08 19:05:07.696472 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:07.696408 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xd9ln" event={"ID":"fd304abd-e907-409f-9faf-94c4b78be9d1","Type":"ContainerStarted","Data":"d45f86c837536bdb60b60e5dd7893a287974fc1fd6d2c93b71359c1fe0c7e8b4"} Jul 08 19:05:07.712950 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:07.712911 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-54hnr" podStartSLOduration=40.38090037 podStartE2EDuration="46.712899023s" totalImagesPullingTime="6.331998653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:04:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:05:07.711872347 +0000 UTC m=+90.970825272" watchObservedRunningTime="2026-07-08 19:05:07.712899023 +0000 UTC m=+90.971851946" Jul 08 19:05:07.730737 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:07.730716 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-54hnr" Jul 08 19:05:07.982688 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:07.982645 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:08.077144 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:08.077091 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:08.252721 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:08.252565 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:08.402319 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:08.402286 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:08.502424 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:08.502380 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:08.603066 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:08.602973 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:08.704154 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:08.704114 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:09.066764 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:09.066715 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:09.611099 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:09.611042 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:09.615441 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:09.614937 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7c4688bdd8-xlx9p" Jul 08 19:05:09.814560 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:09.814520 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:09.915353 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:09.915310 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:10.353006 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:10.352978 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:10.457479 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:10.457439 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:10.710429 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:10.710383 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xd9ln" event={"ID":"fd304abd-e907-409f-9faf-94c4b78be9d1","Type":"ContainerStarted","Data":"ae6939174c2b096984580393958656de25215b848f2c7c31a865c8a943d1bd1e"} Jul 08 19:05:10.735022 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:10.734975 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xd9ln" podStartSLOduration=33.527565188 podStartE2EDuration="40.734955152s" totalImagesPullingTime="7.207389964s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:04:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:05:10.734138104 +0000 UTC m=+93.993091055" watchObservedRunningTime="2026-07-08 19:05:10.734955152 +0000 UTC m=+93.993908074" Jul 08 19:05:10.818810 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:10.818776 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:10.920062 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:10.920022 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:12.499079 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:12.499040 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:12.717197 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:12.717161 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:12.777521 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:12.777441 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-z4mqp"] Jul 08 19:05:12.779761 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:12.779745 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-z4mqp" Jul 08 19:05:12.782006 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:12.781989 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-h87lk\"" Jul 08 19:05:12.783016 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:12.782992 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 08 19:05:12.783139 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:12.783017 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 08 19:05:12.783139 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:12.783030 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 08 19:05:12.783139 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:12.783024 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 08 19:05:12.799811 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:12.799785 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:12.851116 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:12.851071 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e4f979e3-704b-4b38-b0fd-765ae6958073-root\") pod \"node-exporter-z4mqp\" (UID: \"e4f979e3-704b-4b38-b0fd-765ae6958073\") " pod="openshift-monitoring/node-exporter-z4mqp" Jul 08 19:05:12.851241 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:12.851126 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e4f979e3-704b-4b38-b0fd-765ae6958073-node-exporter-tls\") pod \"node-exporter-z4mqp\" (UID: \"e4f979e3-704b-4b38-b0fd-765ae6958073\") " pod="openshift-monitoring/node-exporter-z4mqp" Jul 08 19:05:12.851241 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:12.851146 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e4f979e3-704b-4b38-b0fd-765ae6958073-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z4mqp\" (UID: \"e4f979e3-704b-4b38-b0fd-765ae6958073\") " pod="openshift-monitoring/node-exporter-z4mqp" Jul 08 19:05:12.851241 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:12.851186 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e4f979e3-704b-4b38-b0fd-765ae6958073-node-exporter-accelerators-collector-config\") pod \"node-exporter-z4mqp\" (UID: \"e4f979e3-704b-4b38-b0fd-765ae6958073\") " pod="openshift-monitoring/node-exporter-z4mqp" Jul 08 19:05:12.851241 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:12.851205 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e4f979e3-704b-4b38-b0fd-765ae6958073-node-exporter-wtmp\") pod \"node-exporter-z4mqp\" (UID: \"e4f979e3-704b-4b38-b0fd-765ae6958073\") " pod="openshift-monitoring/node-exporter-z4mqp" Jul 08 19:05:12.851384 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:12.851244 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjdpb\" (UniqueName: \"kubernetes.io/projected/e4f979e3-704b-4b38-b0fd-765ae6958073-kube-api-access-pjdpb\") pod \"node-exporter-z4mqp\" (UID: \"e4f979e3-704b-4b38-b0fd-765ae6958073\") " pod="openshift-monitoring/node-exporter-z4mqp" Jul 08 19:05:12.851384 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:12.851297 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e4f979e3-704b-4b38-b0fd-765ae6958073-sys\") pod \"node-exporter-z4mqp\" (UID: \"e4f979e3-704b-4b38-b0fd-765ae6958073\") " pod="openshift-monitoring/node-exporter-z4mqp" Jul 08 19:05:12.851384 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:12.851359 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4f979e3-704b-4b38-b0fd-765ae6958073-metrics-client-ca\") pod \"node-exporter-z4mqp\" (UID: \"e4f979e3-704b-4b38-b0fd-765ae6958073\") " pod="openshift-monitoring/node-exporter-z4mqp" Jul 08 19:05:12.851468 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:12.851393 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e4f979e3-704b-4b38-b0fd-765ae6958073-node-exporter-textfile\") pod \"node-exporter-z4mqp\" (UID: \"e4f979e3-704b-4b38-b0fd-765ae6958073\") " pod="openshift-monitoring/node-exporter-z4mqp" Jul 08 19:05:12.952346 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:12.952314 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e4f979e3-704b-4b38-b0fd-765ae6958073-sys\") pod \"node-exporter-z4mqp\" (UID: \"e4f979e3-704b-4b38-b0fd-765ae6958073\") " pod="openshift-monitoring/node-exporter-z4mqp" Jul 08 19:05:12.952508 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:12.952352 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4f979e3-704b-4b38-b0fd-765ae6958073-metrics-client-ca\") pod \"node-exporter-z4mqp\" (UID: \"e4f979e3-704b-4b38-b0fd-765ae6958073\") " pod="openshift-monitoring/node-exporter-z4mqp" Jul 08 19:05:12.952508 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:12.952375 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e4f979e3-704b-4b38-b0fd-765ae6958073-node-exporter-textfile\") pod \"node-exporter-z4mqp\" (UID: \"e4f979e3-704b-4b38-b0fd-765ae6958073\") " pod="openshift-monitoring/node-exporter-z4mqp" Jul 08 19:05:12.952508 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:12.952400 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e4f979e3-704b-4b38-b0fd-765ae6958073-root\") pod \"node-exporter-z4mqp\" (UID: \"e4f979e3-704b-4b38-b0fd-765ae6958073\") " pod="openshift-monitoring/node-exporter-z4mqp" Jul 08 19:05:12.952508 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:12.952424 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e4f979e3-704b-4b38-b0fd-765ae6958073-node-exporter-tls\") pod \"node-exporter-z4mqp\" (UID: \"e4f979e3-704b-4b38-b0fd-765ae6958073\") " pod="openshift-monitoring/node-exporter-z4mqp" Jul 08 19:05:12.952508 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:12.952434 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e4f979e3-704b-4b38-b0fd-765ae6958073-sys\") pod \"node-exporter-z4mqp\" (UID: \"e4f979e3-704b-4b38-b0fd-765ae6958073\") " pod="openshift-monitoring/node-exporter-z4mqp" Jul 08 19:05:12.952508 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:12.952440 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e4f979e3-704b-4b38-b0fd-765ae6958073-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z4mqp\" (UID: \"e4f979e3-704b-4b38-b0fd-765ae6958073\") " pod="openshift-monitoring/node-exporter-z4mqp" Jul 08 19:05:12.952818 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:12.952510 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e4f979e3-704b-4b38-b0fd-765ae6958073-node-exporter-accelerators-collector-config\") pod \"node-exporter-z4mqp\" (UID: \"e4f979e3-704b-4b38-b0fd-765ae6958073\") " pod="openshift-monitoring/node-exporter-z4mqp" Jul 08 19:05:12.952818 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:12.952537 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e4f979e3-704b-4b38-b0fd-765ae6958073-root\") pod \"node-exporter-z4mqp\" (UID: \"e4f979e3-704b-4b38-b0fd-765ae6958073\") " pod="openshift-monitoring/node-exporter-z4mqp" Jul 08 19:05:12.952818 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:12.952540 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e4f979e3-704b-4b38-b0fd-765ae6958073-node-exporter-wtmp\") pod \"node-exporter-z4mqp\" (UID: \"e4f979e3-704b-4b38-b0fd-765ae6958073\") " pod="openshift-monitoring/node-exporter-z4mqp" Jul 08 19:05:12.952818 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:12.952637 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjdpb\" (UniqueName: \"kubernetes.io/projected/e4f979e3-704b-4b38-b0fd-765ae6958073-kube-api-access-pjdpb\") pod \"node-exporter-z4mqp\" (UID: \"e4f979e3-704b-4b38-b0fd-765ae6958073\") " pod="openshift-monitoring/node-exporter-z4mqp" Jul 08 19:05:12.952818 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:05:12.952550 2558 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 08 19:05:12.952818 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:05:12.952742 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4f979e3-704b-4b38-b0fd-765ae6958073-node-exporter-tls podName:e4f979e3-704b-4b38-b0fd-765ae6958073 nodeName:}" failed. No retries permitted until 2026-07-08 19:05:13.452719569 +0000 UTC m=+96.711672492 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/e4f979e3-704b-4b38-b0fd-765ae6958073-node-exporter-tls") pod "node-exporter-z4mqp" (UID: "e4f979e3-704b-4b38-b0fd-765ae6958073") : secret "node-exporter-tls" not found Jul 08 19:05:12.953083 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:12.952832 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e4f979e3-704b-4b38-b0fd-765ae6958073-node-exporter-textfile\") pod \"node-exporter-z4mqp\" (UID: \"e4f979e3-704b-4b38-b0fd-765ae6958073\") " pod="openshift-monitoring/node-exporter-z4mqp" Jul 08 19:05:12.953083 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:12.952836 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e4f979e3-704b-4b38-b0fd-765ae6958073-node-exporter-wtmp\") pod \"node-exporter-z4mqp\" (UID: \"e4f979e3-704b-4b38-b0fd-765ae6958073\") " pod="openshift-monitoring/node-exporter-z4mqp" Jul 08 19:05:12.953083 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:12.953001 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4f979e3-704b-4b38-b0fd-765ae6958073-metrics-client-ca\") pod \"node-exporter-z4mqp\" (UID: \"e4f979e3-704b-4b38-b0fd-765ae6958073\") " pod="openshift-monitoring/node-exporter-z4mqp" Jul 08 19:05:12.953083 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:12.953029 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e4f979e3-704b-4b38-b0fd-765ae6958073-node-exporter-accelerators-collector-config\") pod \"node-exporter-z4mqp\" (UID: \"e4f979e3-704b-4b38-b0fd-765ae6958073\") " pod="openshift-monitoring/node-exporter-z4mqp" Jul 08 19:05:12.954675 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:12.954653 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e4f979e3-704b-4b38-b0fd-765ae6958073-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z4mqp\" (UID: \"e4f979e3-704b-4b38-b0fd-765ae6958073\") " pod="openshift-monitoring/node-exporter-z4mqp" Jul 08 19:05:12.967964 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:12.967939 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjdpb\" (UniqueName: \"kubernetes.io/projected/e4f979e3-704b-4b38-b0fd-765ae6958073-kube-api-access-pjdpb\") pod \"node-exporter-z4mqp\" (UID: \"e4f979e3-704b-4b38-b0fd-765ae6958073\") " pod="openshift-monitoring/node-exporter-z4mqp" Jul 08 19:05:13.022264 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:13.022234 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:13.126255 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:13.126216 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:13.225939 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:13.222657 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:13.457328 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:13.457242 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e4f979e3-704b-4b38-b0fd-765ae6958073-node-exporter-tls\") pod \"node-exporter-z4mqp\" (UID: \"e4f979e3-704b-4b38-b0fd-765ae6958073\") " pod="openshift-monitoring/node-exporter-z4mqp" Jul 08 19:05:13.459480 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:13.459449 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e4f979e3-704b-4b38-b0fd-765ae6958073-node-exporter-tls\") pod \"node-exporter-z4mqp\" (UID: \"e4f979e3-704b-4b38-b0fd-765ae6958073\") " pod="openshift-monitoring/node-exporter-z4mqp" Jul 08 19:05:13.689659 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:13.689628 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-z4mqp" Jul 08 19:05:13.700068 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:05:13.700042 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4f979e3_704b_4b38_b0fd_765ae6958073.slice/crio-6c4ce903f9fd6399b7865bf45c3e9707ce7bd52e3e65987c20be6c9d18baac42 WatchSource:0}: Error finding container 6c4ce903f9fd6399b7865bf45c3e9707ce7bd52e3e65987c20be6c9d18baac42: Status 404 returned error can't find the container with id 6c4ce903f9fd6399b7865bf45c3e9707ce7bd52e3e65987c20be6c9d18baac42 Jul 08 19:05:13.722322 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:13.722249 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z4mqp" event={"ID":"e4f979e3-704b-4b38-b0fd-765ae6958073","Type":"ContainerStarted","Data":"6c4ce903f9fd6399b7865bf45c3e9707ce7bd52e3e65987c20be6c9d18baac42"} Jul 08 19:05:13.768167 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:13.768136 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:13.939428 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:13.939393 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:14.242958 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:14.242922 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:14.656222 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:14.656161 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:15.584541 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:15.584513 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-26v6h" Jul 08 19:05:15.707957 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:15.707922 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:15.731397 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:15.731366 2558 generic.go:356] "Generic (PLEG): container finished" podID="e4f979e3-704b-4b38-b0fd-765ae6958073" containerID="d0aa0b765be33d9203d25a5815f8097a1907499c6004a3c5c465c0ee49627e78" exitCode=0 Jul 08 19:05:15.731555 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:15.731437 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z4mqp" event={"ID":"e4f979e3-704b-4b38-b0fd-765ae6958073","Type":"ContainerDied","Data":"d0aa0b765be33d9203d25a5815f8097a1907499c6004a3c5c465c0ee49627e78"} Jul 08 19:05:15.806704 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:15.806676 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:16.027866 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:16.027820 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:16.156721 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:16.156692 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:16.508032 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:16.507954 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c4688bdd8-xlx9p"] Jul 08 19:05:16.584407 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:16.584371 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:16.685682 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:16.685645 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:16.737623 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:16.737563 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z4mqp" event={"ID":"e4f979e3-704b-4b38-b0fd-765ae6958073","Type":"ContainerStarted","Data":"f9a68162cdaa33879bf46d2fe37a9bd01e810be295e570b08272adcabe289304"} Jul 08 19:05:16.737780 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:16.737652 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z4mqp" event={"ID":"e4f979e3-704b-4b38-b0fd-765ae6958073","Type":"ContainerStarted","Data":"d6efbd954532bb439051261124faea4d9bdac9919b4d969a2fa91bc59ffa9161"} Jul 08 19:05:16.757432 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:16.757381 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-z4mqp" podStartSLOduration=3.456382843 podStartE2EDuration="4.75736485s" totalImagesPullingTime="1.23737175s" totalInitContainerRuntime="63.610257ms" isStatefulPod=true podCreationTimestamp="2026-07-08 19:05:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:05:16.756499743 +0000 UTC m=+100.015452668" watchObservedRunningTime="2026-07-08 19:05:16.75736485 +0000 UTC m=+100.016317765" Jul 08 19:05:16.765735 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:16.765643 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:17.470289 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:17.470252 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:17.571076 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:17.571047 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:17.775331 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:17.775242 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:17.932181 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:17.932137 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:19.009891 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:19.009842 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:20.223656 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:20.223617 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:20.227119 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:20.227094 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-c848765-t4frh"] Jul 08 19:05:20.240659 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:20.240637 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c848765-t4frh" Jul 08 19:05:20.241666 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:20.241645 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c848765-t4frh"] Jul 08 19:05:20.243111 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:20.243096 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 08 19:05:20.243228 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:20.243208 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 08 19:05:20.243470 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:20.243453 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 08 19:05:20.244091 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:20.244061 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 08 19:05:20.244091 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:20.244077 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 08 19:05:20.244261 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:20.244101 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 08 19:05:20.244261 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:20.244217 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-tk955\"" Jul 08 19:05:20.244261 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:20.244235 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 08 19:05:20.250368 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:20.250349 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 08 19:05:20.311922 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:20.311895 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/11cb5024-d26a-40f6-a00e-d47470ece773-oauth-serving-cert\") pod \"console-c848765-t4frh\" (UID: \"11cb5024-d26a-40f6-a00e-d47470ece773\") " pod="openshift-console/console-c848765-t4frh" Jul 08 19:05:20.312121 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:20.311932 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/11cb5024-d26a-40f6-a00e-d47470ece773-console-config\") pod \"console-c848765-t4frh\" (UID: \"11cb5024-d26a-40f6-a00e-d47470ece773\") " pod="openshift-console/console-c848765-t4frh" Jul 08 19:05:20.312121 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:20.311949 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/11cb5024-d26a-40f6-a00e-d47470ece773-service-ca\") pod \"console-c848765-t4frh\" (UID: \"11cb5024-d26a-40f6-a00e-d47470ece773\") " pod="openshift-console/console-c848765-t4frh" Jul 08 19:05:20.312121 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:20.311966 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/11cb5024-d26a-40f6-a00e-d47470ece773-console-oauth-config\") pod \"console-c848765-t4frh\" (UID: \"11cb5024-d26a-40f6-a00e-d47470ece773\") " pod="openshift-console/console-c848765-t4frh" Jul 08 19:05:20.312121 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:20.311993 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11cb5024-d26a-40f6-a00e-d47470ece773-trusted-ca-bundle\") pod \"console-c848765-t4frh\" (UID: \"11cb5024-d26a-40f6-a00e-d47470ece773\") " pod="openshift-console/console-c848765-t4frh" Jul 08 19:05:20.312121 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:20.312068 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/11cb5024-d26a-40f6-a00e-d47470ece773-console-serving-cert\") pod \"console-c848765-t4frh\" (UID: \"11cb5024-d26a-40f6-a00e-d47470ece773\") " pod="openshift-console/console-c848765-t4frh" Jul 08 19:05:20.312121 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:20.312099 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlhvp\" (UniqueName: \"kubernetes.io/projected/11cb5024-d26a-40f6-a00e-d47470ece773-kube-api-access-rlhvp\") pod \"console-c848765-t4frh\" (UID: \"11cb5024-d26a-40f6-a00e-d47470ece773\") " pod="openshift-console/console-c848765-t4frh" Jul 08 19:05:20.413030 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:20.413004 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/11cb5024-d26a-40f6-a00e-d47470ece773-console-serving-cert\") pod \"console-c848765-t4frh\" (UID: \"11cb5024-d26a-40f6-a00e-d47470ece773\") " pod="openshift-console/console-c848765-t4frh" Jul 08 19:05:20.413140 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:20.413044 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlhvp\" (UniqueName: \"kubernetes.io/projected/11cb5024-d26a-40f6-a00e-d47470ece773-kube-api-access-rlhvp\") pod \"console-c848765-t4frh\" (UID: \"11cb5024-d26a-40f6-a00e-d47470ece773\") " pod="openshift-console/console-c848765-t4frh" Jul 08 19:05:20.413140 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:20.413075 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/11cb5024-d26a-40f6-a00e-d47470ece773-oauth-serving-cert\") pod \"console-c848765-t4frh\" (UID: \"11cb5024-d26a-40f6-a00e-d47470ece773\") " pod="openshift-console/console-c848765-t4frh" Jul 08 19:05:20.413140 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:20.413106 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/11cb5024-d26a-40f6-a00e-d47470ece773-console-config\") pod \"console-c848765-t4frh\" (UID: \"11cb5024-d26a-40f6-a00e-d47470ece773\") " pod="openshift-console/console-c848765-t4frh" Jul 08 19:05:20.413140 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:20.413132 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/11cb5024-d26a-40f6-a00e-d47470ece773-service-ca\") pod \"console-c848765-t4frh\" (UID: \"11cb5024-d26a-40f6-a00e-d47470ece773\") " pod="openshift-console/console-c848765-t4frh" Jul 08 19:05:20.413302 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:20.413156 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/11cb5024-d26a-40f6-a00e-d47470ece773-console-oauth-config\") pod \"console-c848765-t4frh\" (UID: \"11cb5024-d26a-40f6-a00e-d47470ece773\") " pod="openshift-console/console-c848765-t4frh" Jul 08 19:05:20.413302 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:20.413180 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11cb5024-d26a-40f6-a00e-d47470ece773-trusted-ca-bundle\") pod \"console-c848765-t4frh\" (UID: \"11cb5024-d26a-40f6-a00e-d47470ece773\") " pod="openshift-console/console-c848765-t4frh" Jul 08 19:05:20.413963 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:20.413924 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/11cb5024-d26a-40f6-a00e-d47470ece773-console-config\") pod \"console-c848765-t4frh\" (UID: \"11cb5024-d26a-40f6-a00e-d47470ece773\") " pod="openshift-console/console-c848765-t4frh" Jul 08 19:05:20.414083 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:20.413970 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/11cb5024-d26a-40f6-a00e-d47470ece773-service-ca\") pod \"console-c848765-t4frh\" (UID: \"11cb5024-d26a-40f6-a00e-d47470ece773\") " pod="openshift-console/console-c848765-t4frh" Jul 08 19:05:20.414083 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:20.414027 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/11cb5024-d26a-40f6-a00e-d47470ece773-oauth-serving-cert\") pod \"console-c848765-t4frh\" (UID: \"11cb5024-d26a-40f6-a00e-d47470ece773\") " pod="openshift-console/console-c848765-t4frh" Jul 08 19:05:20.414213 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:20.414097 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11cb5024-d26a-40f6-a00e-d47470ece773-trusted-ca-bundle\") pod \"console-c848765-t4frh\" (UID: \"11cb5024-d26a-40f6-a00e-d47470ece773\") " pod="openshift-console/console-c848765-t4frh" Jul 08 19:05:20.415642 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:20.415615 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/11cb5024-d26a-40f6-a00e-d47470ece773-console-serving-cert\") pod \"console-c848765-t4frh\" (UID: \"11cb5024-d26a-40f6-a00e-d47470ece773\") " pod="openshift-console/console-c848765-t4frh" Jul 08 19:05:20.415974 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:20.415957 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/11cb5024-d26a-40f6-a00e-d47470ece773-console-oauth-config\") pod \"console-c848765-t4frh\" (UID: \"11cb5024-d26a-40f6-a00e-d47470ece773\") " pod="openshift-console/console-c848765-t4frh" Jul 08 19:05:20.420874 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:20.420850 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlhvp\" (UniqueName: \"kubernetes.io/projected/11cb5024-d26a-40f6-a00e-d47470ece773-kube-api-access-rlhvp\") pod \"console-c848765-t4frh\" (UID: \"11cb5024-d26a-40f6-a00e-d47470ece773\") " pod="openshift-console/console-c848765-t4frh" Jul 08 19:05:20.551617 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:20.551529 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c848765-t4frh" Jul 08 19:05:20.675639 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:20.675597 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c848765-t4frh"] Jul 08 19:05:20.678404 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:05:20.678378 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11cb5024_d26a_40f6_a00e_d47470ece773.slice/crio-330d01e0434c1e505e7e8e10026a520d1aa1538696cc83804510d04ece17fcc5 WatchSource:0}: Error finding container 330d01e0434c1e505e7e8e10026a520d1aa1538696cc83804510d04ece17fcc5: Status 404 returned error can't find the container with id 330d01e0434c1e505e7e8e10026a520d1aa1538696cc83804510d04ece17fcc5 Jul 08 19:05:20.752873 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:20.752838 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c848765-t4frh" event={"ID":"11cb5024-d26a-40f6-a00e-d47470ece773","Type":"ContainerStarted","Data":"330d01e0434c1e505e7e8e10026a520d1aa1538696cc83804510d04ece17fcc5"} Jul 08 19:05:21.258383 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:21.258352 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:21.358875 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:21.358839 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:23.767596 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:23.767488 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c848765-t4frh" event={"ID":"11cb5024-d26a-40f6-a00e-d47470ece773","Type":"ContainerStarted","Data":"9e78dad8fb5893c01c2a54e9d494fa75c97e5fa48da2282ec54cdfb0bbd31ae4"} Jul 08 19:05:23.787308 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:23.787256 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-c848765-t4frh" podStartSLOduration=0.976789355 podStartE2EDuration="3.787241352s" totalImagesPullingTime="2.810451997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:05:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:05:23.786464369 +0000 UTC m=+107.045417294" watchObservedRunningTime="2026-07-08 19:05:23.787241352 +0000 UTC m=+107.046194276" Jul 08 19:05:24.837151 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:24.837110 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:24.935971 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:24.935929 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:25.155336 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:25.155285 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:25.257627 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:25.257567 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:25.733596 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:25.733539 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:25.835767 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:25.835733 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:27.007709 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:27.007674 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:27.105055 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:27.105023 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:27.505694 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:27.505659 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:27.606888 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:27.606849 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:29.843906 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:29.843863 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:30.350747 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:30.350704 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:30.413472 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:30.413437 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:30.551913 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:30.551877 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c848765-t4frh" Jul 08 19:05:30.552082 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:30.551955 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c848765-t4frh" Jul 08 19:05:30.556689 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:30.556662 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c848765-t4frh" Jul 08 19:05:30.798750 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:30.798721 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c848765-t4frh" Jul 08 19:05:35.160733 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:35.160689 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:35.280747 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:35.280709 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:35.381371 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:35.381340 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:41.527877 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:41.527812 2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7c4688bdd8-xlx9p" podUID="fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7" containerName="registry" containerID="cri-o://23030c088d7a62df04cd0775a36580feafab4fc9b55ca63d6cc6c2b173dc1082" gracePeriod=30 Jul 08 19:05:41.763838 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:41.763810 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c4688bdd8-xlx9p" Jul 08 19:05:41.841537 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:41.841458 2558 generic.go:356] "Generic (PLEG): container finished" podID="fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7" containerID="23030c088d7a62df04cd0775a36580feafab4fc9b55ca63d6cc6c2b173dc1082" exitCode=0 Jul 08 19:05:41.841537 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:41.841504 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c4688bdd8-xlx9p" Jul 08 19:05:41.841771 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:41.841539 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c4688bdd8-xlx9p" event={"ID":"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7","Type":"ContainerDied","Data":"23030c088d7a62df04cd0775a36580feafab4fc9b55ca63d6cc6c2b173dc1082"} Jul 08 19:05:41.841771 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:41.841621 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c4688bdd8-xlx9p" event={"ID":"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7","Type":"ContainerDied","Data":"2bd365595046a79bfe202ef29774895802323d407c120a070675fd50a71dbd2f"} Jul 08 19:05:41.841771 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:41.841655 2558 scope.go:117] "RemoveContainer" containerID="23030c088d7a62df04cd0775a36580feafab4fc9b55ca63d6cc6c2b173dc1082" Jul 08 19:05:41.851179 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:41.851148 2558 scope.go:117] "RemoveContainer" containerID="23030c088d7a62df04cd0775a36580feafab4fc9b55ca63d6cc6c2b173dc1082" Jul 08 19:05:41.851459 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:05:41.851442 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23030c088d7a62df04cd0775a36580feafab4fc9b55ca63d6cc6c2b173dc1082\": container with ID starting with 23030c088d7a62df04cd0775a36580feafab4fc9b55ca63d6cc6c2b173dc1082 not found: ID does not exist" containerID="23030c088d7a62df04cd0775a36580feafab4fc9b55ca63d6cc6c2b173dc1082" Jul 08 19:05:41.851512 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:41.851466 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23030c088d7a62df04cd0775a36580feafab4fc9b55ca63d6cc6c2b173dc1082"} err="failed to get container status \"23030c088d7a62df04cd0775a36580feafab4fc9b55ca63d6cc6c2b173dc1082\": rpc error: code = NotFound desc = could not find container \"23030c088d7a62df04cd0775a36580feafab4fc9b55ca63d6cc6c2b173dc1082\": container with ID starting with 23030c088d7a62df04cd0775a36580feafab4fc9b55ca63d6cc6c2b173dc1082 not found: ID does not exist" Jul 08 19:05:41.901784 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:41.901758 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-installation-pull-secrets\") pod \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\" (UID: \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\") " Jul 08 19:05:41.901894 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:41.901793 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-registry-certificates\") pod \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\" (UID: \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\") " Jul 08 19:05:41.901894 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:41.901821 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-registry-tls\") pod \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\" (UID: \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\") " Jul 08 19:05:41.901894 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:41.901853 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-bound-sa-token\") pod \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\" (UID: \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\") " Jul 08 19:05:41.901894 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:41.901881 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-image-registry-private-configuration\") pod \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\" (UID: \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\") " Jul 08 19:05:41.902102 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:41.901920 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-ca-trust-extracted\") pod \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\" (UID: \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\") " Jul 08 19:05:41.902102 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:41.901948 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-trusted-ca\") pod \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\" (UID: \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\") " Jul 08 19:05:41.902102 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:41.902039 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwcbw\" (UniqueName: \"kubernetes.io/projected/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-kube-api-access-jwcbw\") pod \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\" (UID: \"fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7\") " Jul 08 19:05:41.902377 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:41.902336 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7" (UID: "fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:05:41.902473 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:41.902411 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7" (UID: "fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:05:41.904626 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:41.904602 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-kube-api-access-jwcbw" (OuterVolumeSpecName: "kube-api-access-jwcbw") pod "fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7" (UID: "fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7"). InnerVolumeSpecName "kube-api-access-jwcbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:05:41.904816 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:41.904777 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7" (UID: "fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:05:41.904816 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:41.904811 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7" (UID: "fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:05:41.904982 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:41.904892 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7" (UID: "fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:05:41.904982 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:41.904925 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7" (UID: "fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:05:41.910992 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:41.910968 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7" (UID: "fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:05:41.983475 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:41.983430 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:42.002915 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:42.002895 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwcbw\" (UniqueName: \"kubernetes.io/projected/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-kube-api-access-jwcbw\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:05:42.003037 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:42.002916 2558 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-installation-pull-secrets\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:05:42.003037 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:42.002930 2558 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-registry-certificates\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:05:42.003037 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:42.002944 2558 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-registry-tls\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:05:42.003037 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:42.002956 2558 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-bound-sa-token\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:05:42.003037 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:42.002968 2558 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-image-registry-private-configuration\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:05:42.003037 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:42.002980 2558 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-ca-trust-extracted\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:05:42.003037 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:42.002992 2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7-trusted-ca\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:05:42.084040 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:42.084002 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:42.166028 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:42.166004 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c4688bdd8-xlx9p"] Jul 08 19:05:42.171368 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:42.171347 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7c4688bdd8-xlx9p"] Jul 08 19:05:43.280858 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:43.280813 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7" path="/var/lib/kubelet/pods/fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7/volumes" Jul 08 19:05:47.248282 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:47.248245 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99034b7e-72e2-4a28-8711-233630b7e9ea-metrics-certs\") pod \"network-metrics-daemon-5rmts\" (UID: \"99034b7e-72e2-4a28-8711-233630b7e9ea\") " pod="openshift-multus/network-metrics-daemon-5rmts" Jul 08 19:05:47.250457 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:47.250438 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99034b7e-72e2-4a28-8711-233630b7e9ea-metrics-certs\") pod \"network-metrics-daemon-5rmts\" (UID: \"99034b7e-72e2-4a28-8711-233630b7e9ea\") " pod="openshift-multus/network-metrics-daemon-5rmts" Jul 08 19:05:47.394997 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:47.394966 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kpjrj\"" Jul 08 19:05:47.402309 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:47.402288 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5rmts" Jul 08 19:05:47.532464 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:47.532436 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5rmts"] Jul 08 19:05:47.534300 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:05:47.534271 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99034b7e_72e2_4a28_8711_233630b7e9ea.slice/crio-de6c88f1eae4950901da22ea1ce135fd684612116a26043d72c4bda02b26fba6 WatchSource:0}: Error finding container de6c88f1eae4950901da22ea1ce135fd684612116a26043d72c4bda02b26fba6: Status 404 returned error can't find the container with id de6c88f1eae4950901da22ea1ce135fd684612116a26043d72c4bda02b26fba6 Jul 08 19:05:47.837351 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:47.837266 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:47.864135 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:47.864095 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5rmts" event={"ID":"99034b7e-72e2-4a28-8711-233630b7e9ea","Type":"ContainerStarted","Data":"de6c88f1eae4950901da22ea1ce135fd684612116a26043d72c4bda02b26fba6"} Jul 08 19:05:47.894783 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:47.894753 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:48.869898 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:48.869854 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5rmts" event={"ID":"99034b7e-72e2-4a28-8711-233630b7e9ea","Type":"ContainerStarted","Data":"43769aba4225f041c92697ce99bcab3cccd64826c9f0e94373c49fa169267e50"} Jul 08 19:05:48.870346 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:48.869920 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5rmts" event={"ID":"99034b7e-72e2-4a28-8711-233630b7e9ea","Type":"ContainerStarted","Data":"a7b4d1df4c9cfb8688eae19d0806cdb345cc26220e4b861c9ccb6fb75c36b6b1"} Jul 08 19:05:48.894564 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:48.894520 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5rmts" podStartSLOduration=130.953105428 podStartE2EDuration="2m11.894507179s" totalImagesPullingTime="941.401751ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:03:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:05:48.892910019 +0000 UTC m=+132.151862945" watchObservedRunningTime="2026-07-08 19:05:48.894507179 +0000 UTC m=+132.153460103" Jul 08 19:05:48.994827 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:48.994796 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:49.094217 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:49.094175 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:49.152344 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:49.152260 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:49.401090 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:49.401044 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:54.894222 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:54.894185 2558 generic.go:356] "Generic (PLEG): container finished" podID="9f2932e3-5510-44d8-bd92-76ebdac8ce3f" containerID="55b9910906c4d27556d04b274773c2e0e473b65382337a5c9ea362d7f289996b" exitCode=0 Jul 08 19:05:54.894598 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:54.894260 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-l4fvs" event={"ID":"9f2932e3-5510-44d8-bd92-76ebdac8ce3f","Type":"ContainerDied","Data":"55b9910906c4d27556d04b274773c2e0e473b65382337a5c9ea362d7f289996b"} Jul 08 19:05:54.894644 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:54.894624 2558 scope.go:117] "RemoveContainer" containerID="55b9910906c4d27556d04b274773c2e0e473b65382337a5c9ea362d7f289996b" Jul 08 19:05:55.900617 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:55.900567 2558 generic.go:356] "Generic (PLEG): container finished" podID="58ce9cbf-4ed4-4482-8786-d92dd2f9a0e0" containerID="8ab04919eaae07e3e7c46a19f2f818dfcf41a57d211eb55bb74f4da26acc6968" exitCode=0 Jul 08 19:05:55.901173 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:55.900641 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-7qzsf" event={"ID":"58ce9cbf-4ed4-4482-8786-d92dd2f9a0e0","Type":"ContainerDied","Data":"8ab04919eaae07e3e7c46a19f2f818dfcf41a57d211eb55bb74f4da26acc6968"} Jul 08 19:05:55.901173 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:55.900995 2558 scope.go:117] "RemoveContainer" containerID="8ab04919eaae07e3e7c46a19f2f818dfcf41a57d211eb55bb74f4da26acc6968" Jul 08 19:05:55.902785 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:55.902767 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-l4fvs" event={"ID":"9f2932e3-5510-44d8-bd92-76ebdac8ce3f","Type":"ContainerStarted","Data":"2858770db84098629dc08059e438e7f96add61c0ef0408e84b5813c5651a4208"} Jul 08 19:05:56.908639 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:56.908602 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-7qzsf" event={"ID":"58ce9cbf-4ed4-4482-8786-d92dd2f9a0e0","Type":"ContainerStarted","Data":"38738a272cc9c3826df0e27ec50a03d418950816a89751af8fd99f390ad736fd"} Jul 08 19:05:56.997971 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:56.997930 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:05:57.096205 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:05:57.096153 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:06:00.924077 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:00.924040 2558 generic.go:356] "Generic (PLEG): container finished" podID="e5f10566-2d90-4c76-846b-8114279ad692" containerID="3e356f0d89b3199deda45bf2d402a44e5069051442a48aa3b16a925e705bb9e7" exitCode=0 Jul 08 19:06:00.924444 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:00.924084 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-fpx7q" event={"ID":"e5f10566-2d90-4c76-846b-8114279ad692","Type":"ContainerDied","Data":"3e356f0d89b3199deda45bf2d402a44e5069051442a48aa3b16a925e705bb9e7"} Jul 08 19:06:00.924444 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:00.924380 2558 scope.go:117] "RemoveContainer" containerID="3e356f0d89b3199deda45bf2d402a44e5069051442a48aa3b16a925e705bb9e7" Jul 08 19:06:01.929496 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:01.929465 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-fpx7q" event={"ID":"e5f10566-2d90-4c76-846b-8114279ad692","Type":"ContainerStarted","Data":"1f3d91634dea620a197b59b7ba6c0a6863067f56df6a7ce6952a9e51b1d9ceba"} Jul 08 19:06:03.662749 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:03.662706 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:06:11.994944 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:11.994902 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:06:12.095406 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:12.095364 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:06:32.201713 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:32.201625 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:06:32.399889 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:32.399846 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:06:41.785970 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:41.785934 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c97dcbfcb-4j5kf"] Jul 08 19:06:41.786443 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:41.786341 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7" containerName="registry" Jul 08 19:06:41.786443 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:41.786357 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7" containerName="registry" Jul 08 19:06:41.786443 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:41.786420 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbb20ea6-abfe-4671-8dc7-4a2e15cef3c7" containerName="registry" Jul 08 19:06:41.789521 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:41.789496 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c97dcbfcb-4j5kf" Jul 08 19:06:41.813128 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:41.813097 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c97dcbfcb-4j5kf"] Jul 08 19:06:41.858092 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:41.858066 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:06:41.875724 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:41.875699 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f8957ad4-270e-424b-b3d7-73d4fd74c922-console-oauth-config\") pod \"console-6c97dcbfcb-4j5kf\" (UID: \"f8957ad4-270e-424b-b3d7-73d4fd74c922\") " pod="openshift-console/console-6c97dcbfcb-4j5kf" Jul 08 19:06:41.875822 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:41.875730 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f8957ad4-270e-424b-b3d7-73d4fd74c922-service-ca\") pod \"console-6c97dcbfcb-4j5kf\" (UID: \"f8957ad4-270e-424b-b3d7-73d4fd74c922\") " pod="openshift-console/console-6c97dcbfcb-4j5kf" Jul 08 19:06:41.875822 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:41.875749 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8957ad4-270e-424b-b3d7-73d4fd74c922-trusted-ca-bundle\") pod \"console-6c97dcbfcb-4j5kf\" (UID: \"f8957ad4-270e-424b-b3d7-73d4fd74c922\") " pod="openshift-console/console-6c97dcbfcb-4j5kf" Jul 08 19:06:41.875890 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:41.875833 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f8957ad4-270e-424b-b3d7-73d4fd74c922-console-serving-cert\") pod \"console-6c97dcbfcb-4j5kf\" (UID: \"f8957ad4-270e-424b-b3d7-73d4fd74c922\") " pod="openshift-console/console-6c97dcbfcb-4j5kf" Jul 08 19:06:41.875890 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:41.875883 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqfks\" (UniqueName: \"kubernetes.io/projected/f8957ad4-270e-424b-b3d7-73d4fd74c922-kube-api-access-hqfks\") pod \"console-6c97dcbfcb-4j5kf\" (UID: \"f8957ad4-270e-424b-b3d7-73d4fd74c922\") " pod="openshift-console/console-6c97dcbfcb-4j5kf" Jul 08 19:06:41.875955 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:41.875905 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f8957ad4-270e-424b-b3d7-73d4fd74c922-oauth-serving-cert\") pod \"console-6c97dcbfcb-4j5kf\" (UID: \"f8957ad4-270e-424b-b3d7-73d4fd74c922\") " pod="openshift-console/console-6c97dcbfcb-4j5kf" Jul 08 19:06:41.875955 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:41.875940 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f8957ad4-270e-424b-b3d7-73d4fd74c922-console-config\") pod \"console-6c97dcbfcb-4j5kf\" (UID: \"f8957ad4-270e-424b-b3d7-73d4fd74c922\") " pod="openshift-console/console-6c97dcbfcb-4j5kf" Jul 08 19:06:41.976425 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:41.976388 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f8957ad4-270e-424b-b3d7-73d4fd74c922-console-serving-cert\") pod \"console-6c97dcbfcb-4j5kf\" (UID: \"f8957ad4-270e-424b-b3d7-73d4fd74c922\") " pod="openshift-console/console-6c97dcbfcb-4j5kf" Jul 08 19:06:41.976547 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:41.976441 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqfks\" (UniqueName: \"kubernetes.io/projected/f8957ad4-270e-424b-b3d7-73d4fd74c922-kube-api-access-hqfks\") pod \"console-6c97dcbfcb-4j5kf\" (UID: \"f8957ad4-270e-424b-b3d7-73d4fd74c922\") " pod="openshift-console/console-6c97dcbfcb-4j5kf" Jul 08 19:06:41.976547 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:41.976464 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f8957ad4-270e-424b-b3d7-73d4fd74c922-oauth-serving-cert\") pod \"console-6c97dcbfcb-4j5kf\" (UID: \"f8957ad4-270e-424b-b3d7-73d4fd74c922\") " pod="openshift-console/console-6c97dcbfcb-4j5kf" Jul 08 19:06:41.976547 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:41.976491 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f8957ad4-270e-424b-b3d7-73d4fd74c922-console-config\") pod \"console-6c97dcbfcb-4j5kf\" (UID: \"f8957ad4-270e-424b-b3d7-73d4fd74c922\") " pod="openshift-console/console-6c97dcbfcb-4j5kf" Jul 08 19:06:41.976547 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:41.976517 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f8957ad4-270e-424b-b3d7-73d4fd74c922-console-oauth-config\") pod \"console-6c97dcbfcb-4j5kf\" (UID: \"f8957ad4-270e-424b-b3d7-73d4fd74c922\") " pod="openshift-console/console-6c97dcbfcb-4j5kf" Jul 08 19:06:41.976547 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:41.976538 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f8957ad4-270e-424b-b3d7-73d4fd74c922-service-ca\") pod \"console-6c97dcbfcb-4j5kf\" (UID: \"f8957ad4-270e-424b-b3d7-73d4fd74c922\") " pod="openshift-console/console-6c97dcbfcb-4j5kf" Jul 08 19:06:41.976796 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:41.976610 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8957ad4-270e-424b-b3d7-73d4fd74c922-trusted-ca-bundle\") pod \"console-6c97dcbfcb-4j5kf\" (UID: \"f8957ad4-270e-424b-b3d7-73d4fd74c922\") " pod="openshift-console/console-6c97dcbfcb-4j5kf" Jul 08 19:06:41.977258 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:41.977234 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f8957ad4-270e-424b-b3d7-73d4fd74c922-service-ca\") pod \"console-6c97dcbfcb-4j5kf\" (UID: \"f8957ad4-270e-424b-b3d7-73d4fd74c922\") " pod="openshift-console/console-6c97dcbfcb-4j5kf" Jul 08 19:06:41.977381 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:41.977360 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f8957ad4-270e-424b-b3d7-73d4fd74c922-oauth-serving-cert\") pod \"console-6c97dcbfcb-4j5kf\" (UID: \"f8957ad4-270e-424b-b3d7-73d4fd74c922\") " pod="openshift-console/console-6c97dcbfcb-4j5kf" Jul 08 19:06:41.977445 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:41.977393 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f8957ad4-270e-424b-b3d7-73d4fd74c922-console-config\") pod \"console-6c97dcbfcb-4j5kf\" (UID: \"f8957ad4-270e-424b-b3d7-73d4fd74c922\") " pod="openshift-console/console-6c97dcbfcb-4j5kf" Jul 08 19:06:41.977785 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:41.977763 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8957ad4-270e-424b-b3d7-73d4fd74c922-trusted-ca-bundle\") pod \"console-6c97dcbfcb-4j5kf\" (UID: \"f8957ad4-270e-424b-b3d7-73d4fd74c922\") " pod="openshift-console/console-6c97dcbfcb-4j5kf" Jul 08 19:06:41.978973 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:41.978954 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f8957ad4-270e-424b-b3d7-73d4fd74c922-console-oauth-config\") pod \"console-6c97dcbfcb-4j5kf\" (UID: \"f8957ad4-270e-424b-b3d7-73d4fd74c922\") " pod="openshift-console/console-6c97dcbfcb-4j5kf" Jul 08 19:06:41.979096 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:41.979078 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f8957ad4-270e-424b-b3d7-73d4fd74c922-console-serving-cert\") pod \"console-6c97dcbfcb-4j5kf\" (UID: \"f8957ad4-270e-424b-b3d7-73d4fd74c922\") " pod="openshift-console/console-6c97dcbfcb-4j5kf" Jul 08 19:06:41.985338 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:41.985320 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqfks\" (UniqueName: \"kubernetes.io/projected/f8957ad4-270e-424b-b3d7-73d4fd74c922-kube-api-access-hqfks\") pod \"console-6c97dcbfcb-4j5kf\" (UID: \"f8957ad4-270e-424b-b3d7-73d4fd74c922\") " pod="openshift-console/console-6c97dcbfcb-4j5kf" Jul 08 19:06:42.100044 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:42.099974 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c97dcbfcb-4j5kf" Jul 08 19:06:42.234150 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:42.234075 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c97dcbfcb-4j5kf"] Jul 08 19:06:42.237040 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:06:42.237015 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8957ad4_270e_424b_b3d7_73d4fd74c922.slice/crio-6c4b8899b3b71b1e1b6d925cd80608299c2c33a740d0209ac3bd86a671029d50 WatchSource:0}: Error finding container 6c4b8899b3b71b1e1b6d925cd80608299c2c33a740d0209ac3bd86a671029d50: Status 404 returned error can't find the container with id 6c4b8899b3b71b1e1b6d925cd80608299c2c33a740d0209ac3bd86a671029d50 Jul 08 19:06:43.081635 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:43.081594 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c97dcbfcb-4j5kf" event={"ID":"f8957ad4-270e-424b-b3d7-73d4fd74c922","Type":"ContainerStarted","Data":"795e4b27becc16669312e665658490aa39550956bb59ba6f2d6089941f430b88"} Jul 08 19:06:43.082055 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:43.081652 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c97dcbfcb-4j5kf" event={"ID":"f8957ad4-270e-424b-b3d7-73d4fd74c922","Type":"ContainerStarted","Data":"6c4b8899b3b71b1e1b6d925cd80608299c2c33a740d0209ac3bd86a671029d50"} Jul 08 19:06:43.107799 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:43.107745 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c97dcbfcb-4j5kf" podStartSLOduration=2.107729465 podStartE2EDuration="2.107729465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:06:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:06:43.102037499 +0000 UTC m=+186.360990421" watchObservedRunningTime="2026-07-08 19:06:43.107729465 +0000 UTC m=+186.366682390" Jul 08 19:06:52.100756 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:52.100719 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c97dcbfcb-4j5kf" Jul 08 19:06:52.101192 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:52.100786 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c97dcbfcb-4j5kf" Jul 08 19:06:52.105531 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:52.105507 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c97dcbfcb-4j5kf" Jul 08 19:06:52.118509 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:52.118486 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c97dcbfcb-4j5kf" Jul 08 19:06:52.224664 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:52.224635 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c848765-t4frh"] Jul 08 19:06:52.728167 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:52.728133 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:06:52.797471 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:06:52.797438 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:07:17.245221 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:17.245177 2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-c848765-t4frh" podUID="11cb5024-d26a-40f6-a00e-d47470ece773" containerName="console" containerID="cri-o://9e78dad8fb5893c01c2a54e9d494fa75c97e5fa48da2282ec54cdfb0bbd31ae4" gracePeriod=15 Jul 08 19:07:17.493093 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:17.493069 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c848765-t4frh" Jul 08 19:07:17.552085 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:17.551999 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/11cb5024-d26a-40f6-a00e-d47470ece773-console-config\") pod \"11cb5024-d26a-40f6-a00e-d47470ece773\" (UID: \"11cb5024-d26a-40f6-a00e-d47470ece773\") " Jul 08 19:07:17.552085 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:17.552054 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rlhvp\" (UniqueName: \"kubernetes.io/projected/11cb5024-d26a-40f6-a00e-d47470ece773-kube-api-access-rlhvp\") pod \"11cb5024-d26a-40f6-a00e-d47470ece773\" (UID: \"11cb5024-d26a-40f6-a00e-d47470ece773\") " Jul 08 19:07:17.552085 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:17.552085 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/11cb5024-d26a-40f6-a00e-d47470ece773-service-ca\") pod \"11cb5024-d26a-40f6-a00e-d47470ece773\" (UID: \"11cb5024-d26a-40f6-a00e-d47470ece773\") " Jul 08 19:07:17.552284 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:17.552148 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/11cb5024-d26a-40f6-a00e-d47470ece773-console-serving-cert\") pod \"11cb5024-d26a-40f6-a00e-d47470ece773\" (UID: \"11cb5024-d26a-40f6-a00e-d47470ece773\") " Jul 08 19:07:17.552284 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:17.552175 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/11cb5024-d26a-40f6-a00e-d47470ece773-oauth-serving-cert\") pod \"11cb5024-d26a-40f6-a00e-d47470ece773\" (UID: \"11cb5024-d26a-40f6-a00e-d47470ece773\") " Jul 08 19:07:17.552416 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:17.552392 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/11cb5024-d26a-40f6-a00e-d47470ece773-console-oauth-config\") pod \"11cb5024-d26a-40f6-a00e-d47470ece773\" (UID: \"11cb5024-d26a-40f6-a00e-d47470ece773\") " Jul 08 19:07:17.552471 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:17.552442 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11cb5024-d26a-40f6-a00e-d47470ece773-trusted-ca-bundle\") pod \"11cb5024-d26a-40f6-a00e-d47470ece773\" (UID: \"11cb5024-d26a-40f6-a00e-d47470ece773\") " Jul 08 19:07:17.552523 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:17.552500 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11cb5024-d26a-40f6-a00e-d47470ece773-console-config" (OuterVolumeSpecName: "console-config") pod "11cb5024-d26a-40f6-a00e-d47470ece773" (UID: "11cb5024-d26a-40f6-a00e-d47470ece773"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:07:17.552607 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:17.552531 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11cb5024-d26a-40f6-a00e-d47470ece773-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "11cb5024-d26a-40f6-a00e-d47470ece773" (UID: "11cb5024-d26a-40f6-a00e-d47470ece773"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:07:17.552668 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:17.552568 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11cb5024-d26a-40f6-a00e-d47470ece773-service-ca" (OuterVolumeSpecName: "service-ca") pod "11cb5024-d26a-40f6-a00e-d47470ece773" (UID: "11cb5024-d26a-40f6-a00e-d47470ece773"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:07:17.552786 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:17.552758 2558 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/11cb5024-d26a-40f6-a00e-d47470ece773-console-config\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:07:17.552786 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:17.552787 2558 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/11cb5024-d26a-40f6-a00e-d47470ece773-service-ca\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:07:17.552953 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:17.552813 2558 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/11cb5024-d26a-40f6-a00e-d47470ece773-oauth-serving-cert\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:07:17.552953 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:17.552872 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11cb5024-d26a-40f6-a00e-d47470ece773-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "11cb5024-d26a-40f6-a00e-d47470ece773" (UID: "11cb5024-d26a-40f6-a00e-d47470ece773"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:07:17.554541 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:17.554518 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11cb5024-d26a-40f6-a00e-d47470ece773-kube-api-access-rlhvp" (OuterVolumeSpecName: "kube-api-access-rlhvp") pod "11cb5024-d26a-40f6-a00e-d47470ece773" (UID: "11cb5024-d26a-40f6-a00e-d47470ece773"). InnerVolumeSpecName "kube-api-access-rlhvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:07:17.554650 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:17.554519 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11cb5024-d26a-40f6-a00e-d47470ece773-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "11cb5024-d26a-40f6-a00e-d47470ece773" (UID: "11cb5024-d26a-40f6-a00e-d47470ece773"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:07:17.554650 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:17.554556 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11cb5024-d26a-40f6-a00e-d47470ece773-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "11cb5024-d26a-40f6-a00e-d47470ece773" (UID: "11cb5024-d26a-40f6-a00e-d47470ece773"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:07:17.654011 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:17.653945 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlhvp\" (UniqueName: \"kubernetes.io/projected/11cb5024-d26a-40f6-a00e-d47470ece773-kube-api-access-rlhvp\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:07:17.654011 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:17.654009 2558 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/11cb5024-d26a-40f6-a00e-d47470ece773-console-serving-cert\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:07:17.654011 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:17.654019 2558 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/11cb5024-d26a-40f6-a00e-d47470ece773-console-oauth-config\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:07:17.654011 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:17.654031 2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11cb5024-d26a-40f6-a00e-d47470ece773-trusted-ca-bundle\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:07:18.208327 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:18.208290 2558 generic.go:356] "Generic (PLEG): container finished" podID="11cb5024-d26a-40f6-a00e-d47470ece773" containerID="9e78dad8fb5893c01c2a54e9d494fa75c97e5fa48da2282ec54cdfb0bbd31ae4" exitCode=0 Jul 08 19:07:18.208487 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:18.208333 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c848765-t4frh" Jul 08 19:07:18.208487 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:18.208361 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c848765-t4frh" event={"ID":"11cb5024-d26a-40f6-a00e-d47470ece773","Type":"ContainerDied","Data":"9e78dad8fb5893c01c2a54e9d494fa75c97e5fa48da2282ec54cdfb0bbd31ae4"} Jul 08 19:07:18.208487 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:18.208402 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c848765-t4frh" event={"ID":"11cb5024-d26a-40f6-a00e-d47470ece773","Type":"ContainerDied","Data":"330d01e0434c1e505e7e8e10026a520d1aa1538696cc83804510d04ece17fcc5"} Jul 08 19:07:18.208487 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:18.208425 2558 scope.go:117] "RemoveContainer" containerID="9e78dad8fb5893c01c2a54e9d494fa75c97e5fa48da2282ec54cdfb0bbd31ae4" Jul 08 19:07:18.218561 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:18.218542 2558 scope.go:117] "RemoveContainer" containerID="9e78dad8fb5893c01c2a54e9d494fa75c97e5fa48da2282ec54cdfb0bbd31ae4" Jul 08 19:07:18.218847 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:07:18.218827 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e78dad8fb5893c01c2a54e9d494fa75c97e5fa48da2282ec54cdfb0bbd31ae4\": container with ID starting with 9e78dad8fb5893c01c2a54e9d494fa75c97e5fa48da2282ec54cdfb0bbd31ae4 not found: ID does not exist" containerID="9e78dad8fb5893c01c2a54e9d494fa75c97e5fa48da2282ec54cdfb0bbd31ae4" Jul 08 19:07:18.218904 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:18.218855 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e78dad8fb5893c01c2a54e9d494fa75c97e5fa48da2282ec54cdfb0bbd31ae4"} err="failed to get container status \"9e78dad8fb5893c01c2a54e9d494fa75c97e5fa48da2282ec54cdfb0bbd31ae4\": rpc error: code = NotFound desc = could not find container \"9e78dad8fb5893c01c2a54e9d494fa75c97e5fa48da2282ec54cdfb0bbd31ae4\": container with ID starting with 9e78dad8fb5893c01c2a54e9d494fa75c97e5fa48da2282ec54cdfb0bbd31ae4 not found: ID does not exist" Jul 08 19:07:18.230236 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:18.230214 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c848765-t4frh"] Jul 08 19:07:18.235727 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:18.235706 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c848765-t4frh"] Jul 08 19:07:19.280066 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:19.280031 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11cb5024-d26a-40f6-a00e-d47470ece773" path="/var/lib/kubelet/pods/11cb5024-d26a-40f6-a00e-d47470ece773/volumes" Jul 08 19:07:44.488187 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:44.488152 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:07:47.530085 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:47.530048 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:07:47.592993 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:47.592960 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:07:50.106489 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:50.106455 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6dfcdb7b79-9trk6"] Jul 08 19:07:50.106960 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:50.106853 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11cb5024-d26a-40f6-a00e-d47470ece773" containerName="console" Jul 08 19:07:50.106960 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:50.106874 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="11cb5024-d26a-40f6-a00e-d47470ece773" containerName="console" Jul 08 19:07:50.106960 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:50.106959 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="11cb5024-d26a-40f6-a00e-d47470ece773" containerName="console" Jul 08 19:07:50.111791 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:50.111700 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dfcdb7b79-9trk6" Jul 08 19:07:50.121451 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:50.121428 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6dfcdb7b79-9trk6"] Jul 08 19:07:50.157117 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:50.157084 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:07:50.203193 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:50.203168 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4693f8b5-7900-4c88-98de-2d26bed5b8c5-console-serving-cert\") pod \"console-6dfcdb7b79-9trk6\" (UID: \"4693f8b5-7900-4c88-98de-2d26bed5b8c5\") " pod="openshift-console/console-6dfcdb7b79-9trk6" Jul 08 19:07:50.203305 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:50.203205 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm4h8\" (UniqueName: \"kubernetes.io/projected/4693f8b5-7900-4c88-98de-2d26bed5b8c5-kube-api-access-cm4h8\") pod \"console-6dfcdb7b79-9trk6\" (UID: \"4693f8b5-7900-4c88-98de-2d26bed5b8c5\") " pod="openshift-console/console-6dfcdb7b79-9trk6" Jul 08 19:07:50.203305 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:50.203235 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4693f8b5-7900-4c88-98de-2d26bed5b8c5-console-oauth-config\") pod \"console-6dfcdb7b79-9trk6\" (UID: \"4693f8b5-7900-4c88-98de-2d26bed5b8c5\") " pod="openshift-console/console-6dfcdb7b79-9trk6" Jul 08 19:07:50.203305 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:50.203256 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4693f8b5-7900-4c88-98de-2d26bed5b8c5-service-ca\") pod \"console-6dfcdb7b79-9trk6\" (UID: \"4693f8b5-7900-4c88-98de-2d26bed5b8c5\") " pod="openshift-console/console-6dfcdb7b79-9trk6" Jul 08 19:07:50.203445 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:50.203318 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4693f8b5-7900-4c88-98de-2d26bed5b8c5-trusted-ca-bundle\") pod \"console-6dfcdb7b79-9trk6\" (UID: \"4693f8b5-7900-4c88-98de-2d26bed5b8c5\") " pod="openshift-console/console-6dfcdb7b79-9trk6" Jul 08 19:07:50.203445 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:50.203347 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4693f8b5-7900-4c88-98de-2d26bed5b8c5-console-config\") pod \"console-6dfcdb7b79-9trk6\" (UID: \"4693f8b5-7900-4c88-98de-2d26bed5b8c5\") " pod="openshift-console/console-6dfcdb7b79-9trk6" Jul 08 19:07:50.203445 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:50.203382 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4693f8b5-7900-4c88-98de-2d26bed5b8c5-oauth-serving-cert\") pod \"console-6dfcdb7b79-9trk6\" (UID: \"4693f8b5-7900-4c88-98de-2d26bed5b8c5\") " pod="openshift-console/console-6dfcdb7b79-9trk6" Jul 08 19:07:50.249514 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:50.249489 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:07:50.304709 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:50.304678 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4693f8b5-7900-4c88-98de-2d26bed5b8c5-console-serving-cert\") pod \"console-6dfcdb7b79-9trk6\" (UID: \"4693f8b5-7900-4c88-98de-2d26bed5b8c5\") " pod="openshift-console/console-6dfcdb7b79-9trk6" Jul 08 19:07:50.304868 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:50.304722 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm4h8\" (UniqueName: \"kubernetes.io/projected/4693f8b5-7900-4c88-98de-2d26bed5b8c5-kube-api-access-cm4h8\") pod \"console-6dfcdb7b79-9trk6\" (UID: \"4693f8b5-7900-4c88-98de-2d26bed5b8c5\") " pod="openshift-console/console-6dfcdb7b79-9trk6" Jul 08 19:07:50.304868 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:50.304746 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4693f8b5-7900-4c88-98de-2d26bed5b8c5-console-oauth-config\") pod \"console-6dfcdb7b79-9trk6\" (UID: \"4693f8b5-7900-4c88-98de-2d26bed5b8c5\") " pod="openshift-console/console-6dfcdb7b79-9trk6" Jul 08 19:07:50.304868 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:50.304761 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4693f8b5-7900-4c88-98de-2d26bed5b8c5-service-ca\") pod \"console-6dfcdb7b79-9trk6\" (UID: \"4693f8b5-7900-4c88-98de-2d26bed5b8c5\") " pod="openshift-console/console-6dfcdb7b79-9trk6" Jul 08 19:07:50.304868 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:50.304789 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4693f8b5-7900-4c88-98de-2d26bed5b8c5-trusted-ca-bundle\") pod \"console-6dfcdb7b79-9trk6\" (UID: \"4693f8b5-7900-4c88-98de-2d26bed5b8c5\") " pod="openshift-console/console-6dfcdb7b79-9trk6" Jul 08 19:07:50.304868 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:50.304813 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4693f8b5-7900-4c88-98de-2d26bed5b8c5-console-config\") pod \"console-6dfcdb7b79-9trk6\" (UID: \"4693f8b5-7900-4c88-98de-2d26bed5b8c5\") " pod="openshift-console/console-6dfcdb7b79-9trk6" Jul 08 19:07:50.304868 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:50.304846 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4693f8b5-7900-4c88-98de-2d26bed5b8c5-oauth-serving-cert\") pod \"console-6dfcdb7b79-9trk6\" (UID: \"4693f8b5-7900-4c88-98de-2d26bed5b8c5\") " pod="openshift-console/console-6dfcdb7b79-9trk6" Jul 08 19:07:50.305646 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:50.305616 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4693f8b5-7900-4c88-98de-2d26bed5b8c5-service-ca\") pod \"console-6dfcdb7b79-9trk6\" (UID: \"4693f8b5-7900-4c88-98de-2d26bed5b8c5\") " pod="openshift-console/console-6dfcdb7b79-9trk6" Jul 08 19:07:50.305774 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:50.305620 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4693f8b5-7900-4c88-98de-2d26bed5b8c5-oauth-serving-cert\") pod \"console-6dfcdb7b79-9trk6\" (UID: \"4693f8b5-7900-4c88-98de-2d26bed5b8c5\") " pod="openshift-console/console-6dfcdb7b79-9trk6" Jul 08 19:07:50.305774 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:50.305656 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4693f8b5-7900-4c88-98de-2d26bed5b8c5-trusted-ca-bundle\") pod \"console-6dfcdb7b79-9trk6\" (UID: \"4693f8b5-7900-4c88-98de-2d26bed5b8c5\") " pod="openshift-console/console-6dfcdb7b79-9trk6" Jul 08 19:07:50.305774 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:50.305719 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4693f8b5-7900-4c88-98de-2d26bed5b8c5-console-config\") pod \"console-6dfcdb7b79-9trk6\" (UID: \"4693f8b5-7900-4c88-98de-2d26bed5b8c5\") " pod="openshift-console/console-6dfcdb7b79-9trk6" Jul 08 19:07:50.307189 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:50.307163 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4693f8b5-7900-4c88-98de-2d26bed5b8c5-console-oauth-config\") pod \"console-6dfcdb7b79-9trk6\" (UID: \"4693f8b5-7900-4c88-98de-2d26bed5b8c5\") " pod="openshift-console/console-6dfcdb7b79-9trk6" Jul 08 19:07:50.307423 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:50.307405 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4693f8b5-7900-4c88-98de-2d26bed5b8c5-console-serving-cert\") pod \"console-6dfcdb7b79-9trk6\" (UID: \"4693f8b5-7900-4c88-98de-2d26bed5b8c5\") " pod="openshift-console/console-6dfcdb7b79-9trk6" Jul 08 19:07:50.313219 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:50.313196 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm4h8\" (UniqueName: \"kubernetes.io/projected/4693f8b5-7900-4c88-98de-2d26bed5b8c5-kube-api-access-cm4h8\") pod \"console-6dfcdb7b79-9trk6\" (UID: \"4693f8b5-7900-4c88-98de-2d26bed5b8c5\") " pod="openshift-console/console-6dfcdb7b79-9trk6" Jul 08 19:07:50.423477 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:50.423449 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dfcdb7b79-9trk6" Jul 08 19:07:50.572649 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:50.572560 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6dfcdb7b79-9trk6"] Jul 08 19:07:50.574920 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:07:50.574893 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4693f8b5_7900_4c88_98de_2d26bed5b8c5.slice/crio-43aa8e213189aca47ea03b1a562d85fff7e2e23d6d254d776558ec4d852f0461 WatchSource:0}: Error finding container 43aa8e213189aca47ea03b1a562d85fff7e2e23d6d254d776558ec4d852f0461: Status 404 returned error can't find the container with id 43aa8e213189aca47ea03b1a562d85fff7e2e23d6d254d776558ec4d852f0461 Jul 08 19:07:51.327788 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:51.327747 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dfcdb7b79-9trk6" event={"ID":"4693f8b5-7900-4c88-98de-2d26bed5b8c5","Type":"ContainerStarted","Data":"8ce269f922ce60c25f3f1d45c2ba1c8bd74632dbf3d754ca9d0009e5c87ce85e"} Jul 08 19:07:51.328146 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:51.327807 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dfcdb7b79-9trk6" event={"ID":"4693f8b5-7900-4c88-98de-2d26bed5b8c5","Type":"ContainerStarted","Data":"43aa8e213189aca47ea03b1a562d85fff7e2e23d6d254d776558ec4d852f0461"} Jul 08 19:07:51.346323 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:51.346284 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6dfcdb7b79-9trk6" podStartSLOduration=1.346266218 podStartE2EDuration="1.346266218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:07:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:07:51.345922044 +0000 UTC m=+254.604874968" watchObservedRunningTime="2026-07-08 19:07:51.346266218 +0000 UTC m=+254.605219141" Jul 08 19:07:59.289085 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:59.289039 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:07:59.356356 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:07:59.356323 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:08:00.423688 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:00.423638 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6dfcdb7b79-9trk6" Jul 08 19:08:00.423688 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:00.423707 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6dfcdb7b79-9trk6" Jul 08 19:08:00.428847 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:00.428823 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6dfcdb7b79-9trk6" Jul 08 19:08:01.369068 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:01.369038 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6dfcdb7b79-9trk6" Jul 08 19:08:01.412505 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:01.412471 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c97dcbfcb-4j5kf"] Jul 08 19:08:01.511500 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:01.511467 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:08:01.568154 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:01.568121 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:08:01.648308 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:01.648210 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:08:04.743799 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:04.743763 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:08:11.991960 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:11.991909 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:08:12.097324 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:12.097284 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:08:12.158139 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:12.158091 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:08:26.439220 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:26.439182 2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6c97dcbfcb-4j5kf" podUID="f8957ad4-270e-424b-b3d7-73d4fd74c922" containerName="console" containerID="cri-o://795e4b27becc16669312e665658490aa39550956bb59ba6f2d6089941f430b88" gracePeriod=15 Jul 08 19:08:26.678804 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:26.678781 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c97dcbfcb-4j5kf" Jul 08 19:08:26.690882 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:26.690826 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f8957ad4-270e-424b-b3d7-73d4fd74c922-console-oauth-config\") pod \"f8957ad4-270e-424b-b3d7-73d4fd74c922\" (UID: \"f8957ad4-270e-424b-b3d7-73d4fd74c922\") " Jul 08 19:08:26.690882 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:26.690866 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f8957ad4-270e-424b-b3d7-73d4fd74c922-console-serving-cert\") pod \"f8957ad4-270e-424b-b3d7-73d4fd74c922\" (UID: \"f8957ad4-270e-424b-b3d7-73d4fd74c922\") " Jul 08 19:08:26.691055 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:26.690894 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f8957ad4-270e-424b-b3d7-73d4fd74c922-oauth-serving-cert\") pod \"f8957ad4-270e-424b-b3d7-73d4fd74c922\" (UID: \"f8957ad4-270e-424b-b3d7-73d4fd74c922\") " Jul 08 19:08:26.691055 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:26.690950 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f8957ad4-270e-424b-b3d7-73d4fd74c922-service-ca\") pod \"f8957ad4-270e-424b-b3d7-73d4fd74c922\" (UID: \"f8957ad4-270e-424b-b3d7-73d4fd74c922\") " Jul 08 19:08:26.691055 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:26.690977 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8957ad4-270e-424b-b3d7-73d4fd74c922-trusted-ca-bundle\") pod \"f8957ad4-270e-424b-b3d7-73d4fd74c922\" (UID: \"f8957ad4-270e-424b-b3d7-73d4fd74c922\") " Jul 08 19:08:26.691055 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:26.691005 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hqfks\" (UniqueName: \"kubernetes.io/projected/f8957ad4-270e-424b-b3d7-73d4fd74c922-kube-api-access-hqfks\") pod \"f8957ad4-270e-424b-b3d7-73d4fd74c922\" (UID: \"f8957ad4-270e-424b-b3d7-73d4fd74c922\") " Jul 08 19:08:26.691055 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:26.691047 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f8957ad4-270e-424b-b3d7-73d4fd74c922-console-config\") pod \"f8957ad4-270e-424b-b3d7-73d4fd74c922\" (UID: \"f8957ad4-270e-424b-b3d7-73d4fd74c922\") " Jul 08 19:08:26.691346 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:26.691319 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8957ad4-270e-424b-b3d7-73d4fd74c922-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f8957ad4-270e-424b-b3d7-73d4fd74c922" (UID: "f8957ad4-270e-424b-b3d7-73d4fd74c922"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:08:26.691471 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:26.691358 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8957ad4-270e-424b-b3d7-73d4fd74c922-service-ca" (OuterVolumeSpecName: "service-ca") pod "f8957ad4-270e-424b-b3d7-73d4fd74c922" (UID: "f8957ad4-270e-424b-b3d7-73d4fd74c922"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:08:26.691471 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:26.691401 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8957ad4-270e-424b-b3d7-73d4fd74c922-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f8957ad4-270e-424b-b3d7-73d4fd74c922" (UID: "f8957ad4-270e-424b-b3d7-73d4fd74c922"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:08:26.691628 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:26.691606 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8957ad4-270e-424b-b3d7-73d4fd74c922-console-config" (OuterVolumeSpecName: "console-config") pod "f8957ad4-270e-424b-b3d7-73d4fd74c922" (UID: "f8957ad4-270e-424b-b3d7-73d4fd74c922"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:08:26.692900 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:26.692872 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8957ad4-270e-424b-b3d7-73d4fd74c922-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f8957ad4-270e-424b-b3d7-73d4fd74c922" (UID: "f8957ad4-270e-424b-b3d7-73d4fd74c922"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:08:26.693060 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:26.693033 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8957ad4-270e-424b-b3d7-73d4fd74c922-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f8957ad4-270e-424b-b3d7-73d4fd74c922" (UID: "f8957ad4-270e-424b-b3d7-73d4fd74c922"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:08:26.693160 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:26.693086 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8957ad4-270e-424b-b3d7-73d4fd74c922-kube-api-access-hqfks" (OuterVolumeSpecName: "kube-api-access-hqfks") pod "f8957ad4-270e-424b-b3d7-73d4fd74c922" (UID: "f8957ad4-270e-424b-b3d7-73d4fd74c922"). InnerVolumeSpecName "kube-api-access-hqfks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:08:26.791835 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:26.791806 2558 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f8957ad4-270e-424b-b3d7-73d4fd74c922-console-config\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:08:26.791835 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:26.791830 2558 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f8957ad4-270e-424b-b3d7-73d4fd74c922-console-oauth-config\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:08:26.791835 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:26.791842 2558 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f8957ad4-270e-424b-b3d7-73d4fd74c922-console-serving-cert\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:08:26.792045 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:26.791851 2558 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f8957ad4-270e-424b-b3d7-73d4fd74c922-oauth-serving-cert\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:08:26.792045 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:26.791859 2558 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f8957ad4-270e-424b-b3d7-73d4fd74c922-service-ca\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:08:26.792045 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:26.791867 2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8957ad4-270e-424b-b3d7-73d4fd74c922-trusted-ca-bundle\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:08:26.792045 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:26.791876 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqfks\" (UniqueName: \"kubernetes.io/projected/f8957ad4-270e-424b-b3d7-73d4fd74c922-kube-api-access-hqfks\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:08:27.458870 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:27.458827 2558 generic.go:356] "Generic (PLEG): container finished" podID="f8957ad4-270e-424b-b3d7-73d4fd74c922" containerID="795e4b27becc16669312e665658490aa39550956bb59ba6f2d6089941f430b88" exitCode=0 Jul 08 19:08:27.459284 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:27.458876 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c97dcbfcb-4j5kf" event={"ID":"f8957ad4-270e-424b-b3d7-73d4fd74c922","Type":"ContainerDied","Data":"795e4b27becc16669312e665658490aa39550956bb59ba6f2d6089941f430b88"} Jul 08 19:08:27.459284 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:27.458881 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c97dcbfcb-4j5kf" Jul 08 19:08:27.459284 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:27.458947 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c97dcbfcb-4j5kf" event={"ID":"f8957ad4-270e-424b-b3d7-73d4fd74c922","Type":"ContainerDied","Data":"6c4b8899b3b71b1e1b6d925cd80608299c2c33a740d0209ac3bd86a671029d50"} Jul 08 19:08:27.459284 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:27.458972 2558 scope.go:117] "RemoveContainer" containerID="795e4b27becc16669312e665658490aa39550956bb59ba6f2d6089941f430b88" Jul 08 19:08:27.466827 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:27.466807 2558 scope.go:117] "RemoveContainer" containerID="795e4b27becc16669312e665658490aa39550956bb59ba6f2d6089941f430b88" Jul 08 19:08:27.467092 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:08:27.467075 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"795e4b27becc16669312e665658490aa39550956bb59ba6f2d6089941f430b88\": container with ID starting with 795e4b27becc16669312e665658490aa39550956bb59ba6f2d6089941f430b88 not found: ID does not exist" containerID="795e4b27becc16669312e665658490aa39550956bb59ba6f2d6089941f430b88" Jul 08 19:08:27.467149 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:27.467099 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"795e4b27becc16669312e665658490aa39550956bb59ba6f2d6089941f430b88"} err="failed to get container status \"795e4b27becc16669312e665658490aa39550956bb59ba6f2d6089941f430b88\": rpc error: code = NotFound desc = could not find container \"795e4b27becc16669312e665658490aa39550956bb59ba6f2d6089941f430b88\": container with ID starting with 795e4b27becc16669312e665658490aa39550956bb59ba6f2d6089941f430b88 not found: ID does not exist" Jul 08 19:08:27.476026 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:27.476000 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c97dcbfcb-4j5kf"] Jul 08 19:08:27.479768 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:27.479749 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c97dcbfcb-4j5kf"] Jul 08 19:08:29.280898 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:29.280863 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8957ad4-270e-424b-b3d7-73d4fd74c922" path="/var/lib/kubelet/pods/f8957ad4-270e-424b-b3d7-73d4fd74c922/volumes" Jul 08 19:08:35.229238 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:35.229204 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:08:35.323254 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:35.323217 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:08:35.949798 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:35.949762 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gkgpz"] Jul 08 19:08:35.950130 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:35.950111 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8957ad4-270e-424b-b3d7-73d4fd74c922" containerName="console" Jul 08 19:08:35.950230 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:35.950132 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8957ad4-270e-424b-b3d7-73d4fd74c922" containerName="console" Jul 08 19:08:35.950287 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:35.950235 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8957ad4-270e-424b-b3d7-73d4fd74c922" containerName="console" Jul 08 19:08:35.953511 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:35.953492 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gkgpz" Jul 08 19:08:35.956697 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:35.956675 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lm5m4\"" Jul 08 19:08:35.956787 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:35.956723 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 08 19:08:35.957459 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:35.957439 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 08 19:08:35.966242 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:35.966225 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gkgpz"] Jul 08 19:08:36.053062 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:36.053024 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af4d37a4-435d-4549-871a-38fc217625c9-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gkgpz\" (UID: \"af4d37a4-435d-4549-871a-38fc217625c9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gkgpz" Jul 08 19:08:36.053223 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:36.053080 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af4d37a4-435d-4549-871a-38fc217625c9-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gkgpz\" (UID: \"af4d37a4-435d-4549-871a-38fc217625c9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gkgpz" Jul 08 19:08:36.053223 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:36.053101 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkws9\" (UniqueName: \"kubernetes.io/projected/af4d37a4-435d-4549-871a-38fc217625c9-kube-api-access-fkws9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gkgpz\" (UID: \"af4d37a4-435d-4549-871a-38fc217625c9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gkgpz" Jul 08 19:08:36.154075 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:36.154042 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af4d37a4-435d-4549-871a-38fc217625c9-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gkgpz\" (UID: \"af4d37a4-435d-4549-871a-38fc217625c9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gkgpz" Jul 08 19:08:36.154204 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:36.154109 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af4d37a4-435d-4549-871a-38fc217625c9-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gkgpz\" (UID: \"af4d37a4-435d-4549-871a-38fc217625c9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gkgpz" Jul 08 19:08:36.154204 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:36.154138 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkws9\" (UniqueName: \"kubernetes.io/projected/af4d37a4-435d-4549-871a-38fc217625c9-kube-api-access-fkws9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gkgpz\" (UID: \"af4d37a4-435d-4549-871a-38fc217625c9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gkgpz" Jul 08 19:08:36.154431 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:36.154408 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af4d37a4-435d-4549-871a-38fc217625c9-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gkgpz\" (UID: \"af4d37a4-435d-4549-871a-38fc217625c9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gkgpz" Jul 08 19:08:36.154497 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:36.154428 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af4d37a4-435d-4549-871a-38fc217625c9-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gkgpz\" (UID: \"af4d37a4-435d-4549-871a-38fc217625c9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gkgpz" Jul 08 19:08:36.163048 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:36.163004 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkws9\" (UniqueName: \"kubernetes.io/projected/af4d37a4-435d-4549-871a-38fc217625c9-kube-api-access-fkws9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gkgpz\" (UID: \"af4d37a4-435d-4549-871a-38fc217625c9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gkgpz" Jul 08 19:08:36.263032 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:36.262971 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gkgpz" Jul 08 19:08:36.390562 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:36.390533 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gkgpz"] Jul 08 19:08:36.393195 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:08:36.393166 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf4d37a4_435d_4549_871a_38fc217625c9.slice/crio-384c21e6fd813f9bf0de8eaa6db0aad890b0332aa85e116c43decfe21d87b4c4 WatchSource:0}: Error finding container 384c21e6fd813f9bf0de8eaa6db0aad890b0332aa85e116c43decfe21d87b4c4: Status 404 returned error can't find the container with id 384c21e6fd813f9bf0de8eaa6db0aad890b0332aa85e116c43decfe21d87b4c4 Jul 08 19:08:36.492466 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:36.492430 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gkgpz" event={"ID":"af4d37a4-435d-4549-871a-38fc217625c9","Type":"ContainerStarted","Data":"384c21e6fd813f9bf0de8eaa6db0aad890b0332aa85e116c43decfe21d87b4c4"} Jul 08 19:08:37.138204 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:37.138177 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-54hnr_1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a/console-operator/2.log" Jul 08 19:08:37.144790 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:37.144765 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-54hnr_1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a/console-operator/2.log" Jul 08 19:08:37.150599 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:37.150558 2558 kubelet.go:1648] "Image garbage collection succeeded" Jul 08 19:08:44.528991 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:44.528951 2558 generic.go:356] "Generic (PLEG): container finished" podID="af4d37a4-435d-4549-871a-38fc217625c9" containerID="d3416e13636a2a958edcf4e4f3d1ed3be34b71458ded4b0112d8465307cb7320" exitCode=0 Jul 08 19:08:44.529347 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:44.529037 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gkgpz" event={"ID":"af4d37a4-435d-4549-871a-38fc217625c9","Type":"ContainerDied","Data":"d3416e13636a2a958edcf4e4f3d1ed3be34b71458ded4b0112d8465307cb7320"} Jul 08 19:08:44.529967 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:44.529952 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 19:08:47.543329 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:47.543298 2558 generic.go:356] "Generic (PLEG): container finished" podID="af4d37a4-435d-4549-871a-38fc217625c9" containerID="3cd03cd698f46ba3c3813fd783abcab186ffec939d042c17f24d4cea29b8f725" exitCode=0 Jul 08 19:08:47.543740 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:47.543337 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gkgpz" event={"ID":"af4d37a4-435d-4549-871a-38fc217625c9","Type":"ContainerDied","Data":"3cd03cd698f46ba3c3813fd783abcab186ffec939d042c17f24d4cea29b8f725"} Jul 08 19:08:54.575140 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:54.575099 2558 generic.go:356] "Generic (PLEG): container finished" podID="af4d37a4-435d-4549-871a-38fc217625c9" containerID="fa3b36cbec25205533217e4e7cfaf96f9772166fe0cad9aa199d0764bdb6b9eb" exitCode=0 Jul 08 19:08:54.575514 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:54.575179 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gkgpz" event={"ID":"af4d37a4-435d-4549-871a-38fc217625c9","Type":"ContainerDied","Data":"fa3b36cbec25205533217e4e7cfaf96f9772166fe0cad9aa199d0764bdb6b9eb"} Jul 08 19:08:55.700707 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:55.700684 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gkgpz" Jul 08 19:08:55.814649 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:55.814614 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af4d37a4-435d-4549-871a-38fc217625c9-util\") pod \"af4d37a4-435d-4549-871a-38fc217625c9\" (UID: \"af4d37a4-435d-4549-871a-38fc217625c9\") " Jul 08 19:08:55.814842 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:55.814701 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkws9\" (UniqueName: \"kubernetes.io/projected/af4d37a4-435d-4549-871a-38fc217625c9-kube-api-access-fkws9\") pod \"af4d37a4-435d-4549-871a-38fc217625c9\" (UID: \"af4d37a4-435d-4549-871a-38fc217625c9\") " Jul 08 19:08:55.814842 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:55.814753 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af4d37a4-435d-4549-871a-38fc217625c9-bundle\") pod \"af4d37a4-435d-4549-871a-38fc217625c9\" (UID: \"af4d37a4-435d-4549-871a-38fc217625c9\") " Jul 08 19:08:55.815404 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:55.815370 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af4d37a4-435d-4549-871a-38fc217625c9-bundle" (OuterVolumeSpecName: "bundle") pod "af4d37a4-435d-4549-871a-38fc217625c9" (UID: "af4d37a4-435d-4549-871a-38fc217625c9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:08:55.816907 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:55.816884 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af4d37a4-435d-4549-871a-38fc217625c9-kube-api-access-fkws9" (OuterVolumeSpecName: "kube-api-access-fkws9") pod "af4d37a4-435d-4549-871a-38fc217625c9" (UID: "af4d37a4-435d-4549-871a-38fc217625c9"). InnerVolumeSpecName "kube-api-access-fkws9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:08:55.819638 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:55.819617 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af4d37a4-435d-4549-871a-38fc217625c9-util" (OuterVolumeSpecName: "util") pod "af4d37a4-435d-4549-871a-38fc217625c9" (UID: "af4d37a4-435d-4549-871a-38fc217625c9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:08:55.915412 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:55.915377 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkws9\" (UniqueName: \"kubernetes.io/projected/af4d37a4-435d-4549-871a-38fc217625c9-kube-api-access-fkws9\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:08:55.915412 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:55.915411 2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af4d37a4-435d-4549-871a-38fc217625c9-bundle\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:08:55.915605 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:55.915423 2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af4d37a4-435d-4549-871a-38fc217625c9-util\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:08:56.584983 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:56.584953 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gkgpz" Jul 08 19:08:56.585164 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:56.584952 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gkgpz" event={"ID":"af4d37a4-435d-4549-871a-38fc217625c9","Type":"ContainerDied","Data":"384c21e6fd813f9bf0de8eaa6db0aad890b0332aa85e116c43decfe21d87b4c4"} Jul 08 19:08:56.585164 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:08:56.585077 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="384c21e6fd813f9bf0de8eaa6db0aad890b0332aa85e116c43decfe21d87b4c4" Jul 08 19:09:02.087379 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:02.087345 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:09:03.280287 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:03.280254 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:09:04.049387 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:04.049351 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m4nlt"] Jul 08 19:09:04.049641 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:04.049628 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af4d37a4-435d-4549-871a-38fc217625c9" containerName="util" Jul 08 19:09:04.049693 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:04.049642 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="af4d37a4-435d-4549-871a-38fc217625c9" containerName="util" Jul 08 19:09:04.049693 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:04.049651 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af4d37a4-435d-4549-871a-38fc217625c9" containerName="pull" Jul 08 19:09:04.049693 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:04.049658 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="af4d37a4-435d-4549-871a-38fc217625c9" containerName="pull" Jul 08 19:09:04.049693 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:04.049668 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af4d37a4-435d-4549-871a-38fc217625c9" containerName="extract" Jul 08 19:09:04.049693 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:04.049674 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="af4d37a4-435d-4549-871a-38fc217625c9" containerName="extract" Jul 08 19:09:04.049836 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:04.049735 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="af4d37a4-435d-4549-871a-38fc217625c9" containerName="extract" Jul 08 19:09:04.051698 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:04.051680 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m4nlt" Jul 08 19:09:04.055877 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:04.055860 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 08 19:09:04.056210 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:04.056196 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 08 19:09:04.056309 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:04.056293 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-tfjjv\"" Jul 08 19:09:04.067981 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:04.067962 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m4nlt"] Jul 08 19:09:04.181897 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:04.181855 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5gnr\" (UniqueName: \"kubernetes.io/projected/9c52bfd1-510b-444e-92bf-2775a5ddea17-kube-api-access-x5gnr\") pod \"cert-manager-operator-controller-manager-55c444bf86-m4nlt\" (UID: \"9c52bfd1-510b-444e-92bf-2775a5ddea17\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m4nlt" Jul 08 19:09:04.182045 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:04.181910 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c52bfd1-510b-444e-92bf-2775a5ddea17-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-m4nlt\" (UID: \"9c52bfd1-510b-444e-92bf-2775a5ddea17\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m4nlt" Jul 08 19:09:04.282415 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:04.282383 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c52bfd1-510b-444e-92bf-2775a5ddea17-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-m4nlt\" (UID: \"9c52bfd1-510b-444e-92bf-2775a5ddea17\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m4nlt" Jul 08 19:09:04.282762 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:04.282438 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5gnr\" (UniqueName: \"kubernetes.io/projected/9c52bfd1-510b-444e-92bf-2775a5ddea17-kube-api-access-x5gnr\") pod \"cert-manager-operator-controller-manager-55c444bf86-m4nlt\" (UID: \"9c52bfd1-510b-444e-92bf-2775a5ddea17\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m4nlt" Jul 08 19:09:04.282857 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:04.282840 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c52bfd1-510b-444e-92bf-2775a5ddea17-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-m4nlt\" (UID: \"9c52bfd1-510b-444e-92bf-2775a5ddea17\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m4nlt" Jul 08 19:09:04.293509 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:04.293485 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5gnr\" (UniqueName: \"kubernetes.io/projected/9c52bfd1-510b-444e-92bf-2775a5ddea17-kube-api-access-x5gnr\") pod \"cert-manager-operator-controller-manager-55c444bf86-m4nlt\" (UID: \"9c52bfd1-510b-444e-92bf-2775a5ddea17\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m4nlt" Jul 08 19:09:04.360494 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:04.360437 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m4nlt" Jul 08 19:09:04.489204 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:04.489180 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m4nlt"] Jul 08 19:09:04.491746 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:09:04.491716 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c52bfd1_510b_444e_92bf_2775a5ddea17.slice/crio-0fb633a32376ae609bc7c3b0a5bdcf2e7d78269c0b35c559d682aa1c0df48097 WatchSource:0}: Error finding container 0fb633a32376ae609bc7c3b0a5bdcf2e7d78269c0b35c559d682aa1c0df48097: Status 404 returned error can't find the container with id 0fb633a32376ae609bc7c3b0a5bdcf2e7d78269c0b35c559d682aa1c0df48097 Jul 08 19:09:04.615408 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:04.615314 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m4nlt" event={"ID":"9c52bfd1-510b-444e-92bf-2775a5ddea17","Type":"ContainerStarted","Data":"0fb633a32376ae609bc7c3b0a5bdcf2e7d78269c0b35c559d682aa1c0df48097"} Jul 08 19:09:05.893521 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:05.893476 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:09:05.952977 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:05.952933 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:09:06.627087 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:06.627055 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m4nlt" event={"ID":"9c52bfd1-510b-444e-92bf-2775a5ddea17","Type":"ContainerStarted","Data":"57e80b81987627dcbbfd0ff9c81123ef08aeadc36f5619410644370e47bd0635"} Jul 08 19:09:06.655447 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:06.655389 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m4nlt" podStartSLOduration=1.033662982 podStartE2EDuration="2.655364973s" totalImagesPullingTime="1.621701991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:09:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:09:06.653464438 +0000 UTC m=+329.912417364" watchObservedRunningTime="2026-07-08 19:09:06.655364973 +0000 UTC m=+329.914317902" Jul 08 19:09:07.118218 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:07.118126 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:09:07.221731 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:07.221695 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:09:07.715544 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:07.715510 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffhnlm"] Jul 08 19:09:07.718410 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:07.718394 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffhnlm" Jul 08 19:09:07.721035 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:07.721007 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 08 19:09:07.722263 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:07.722240 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 08 19:09:07.722374 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:07.722356 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lm5m4\"" Jul 08 19:09:07.728251 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:07.728225 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffhnlm"] Jul 08 19:09:07.812630 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:07.812597 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fc24815-f5ae-4e3f-8cca-6637468415a2-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffhnlm\" (UID: \"0fc24815-f5ae-4e3f-8cca-6637468415a2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffhnlm" Jul 08 19:09:07.812774 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:07.812640 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fc24815-f5ae-4e3f-8cca-6637468415a2-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffhnlm\" (UID: \"0fc24815-f5ae-4e3f-8cca-6637468415a2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffhnlm" Jul 08 19:09:07.812774 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:07.812704 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbsrv\" (UniqueName: \"kubernetes.io/projected/0fc24815-f5ae-4e3f-8cca-6637468415a2-kube-api-access-hbsrv\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffhnlm\" (UID: \"0fc24815-f5ae-4e3f-8cca-6637468415a2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffhnlm" Jul 08 19:09:07.833385 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:07.833347 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:09:07.913345 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:07.913316 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbsrv\" (UniqueName: \"kubernetes.io/projected/0fc24815-f5ae-4e3f-8cca-6637468415a2-kube-api-access-hbsrv\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffhnlm\" (UID: \"0fc24815-f5ae-4e3f-8cca-6637468415a2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffhnlm" Jul 08 19:09:07.913505 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:07.913362 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fc24815-f5ae-4e3f-8cca-6637468415a2-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffhnlm\" (UID: \"0fc24815-f5ae-4e3f-8cca-6637468415a2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffhnlm" Jul 08 19:09:07.913505 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:07.913404 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fc24815-f5ae-4e3f-8cca-6637468415a2-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffhnlm\" (UID: \"0fc24815-f5ae-4e3f-8cca-6637468415a2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffhnlm" Jul 08 19:09:07.913811 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:07.913794 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fc24815-f5ae-4e3f-8cca-6637468415a2-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffhnlm\" (UID: \"0fc24815-f5ae-4e3f-8cca-6637468415a2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffhnlm" Jul 08 19:09:07.913874 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:07.913852 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fc24815-f5ae-4e3f-8cca-6637468415a2-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffhnlm\" (UID: \"0fc24815-f5ae-4e3f-8cca-6637468415a2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffhnlm" Jul 08 19:09:07.924794 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:07.924766 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbsrv\" (UniqueName: \"kubernetes.io/projected/0fc24815-f5ae-4e3f-8cca-6637468415a2-kube-api-access-hbsrv\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffhnlm\" (UID: \"0fc24815-f5ae-4e3f-8cca-6637468415a2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffhnlm" Jul 08 19:09:07.929986 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:07.929920 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:09:08.028204 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:08.028120 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffhnlm" Jul 08 19:09:08.157535 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:08.157504 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffhnlm"] Jul 08 19:09:08.160761 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:09:08.160725 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fc24815_f5ae_4e3f_8cca_6637468415a2.slice/crio-b8a3799caedbb3fe6074913affccfad4c71ccf5362799501dfd5070028860cf7 WatchSource:0}: Error finding container b8a3799caedbb3fe6074913affccfad4c71ccf5362799501dfd5070028860cf7: Status 404 returned error can't find the container with id b8a3799caedbb3fe6074913affccfad4c71ccf5362799501dfd5070028860cf7 Jul 08 19:09:08.637693 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:08.637661 2558 generic.go:356] "Generic (PLEG): container finished" podID="0fc24815-f5ae-4e3f-8cca-6637468415a2" containerID="564f34874888e87412f66fb89ffa74e80f3727fd84642a6cdadda8ea76f92a3b" exitCode=0 Jul 08 19:09:08.637861 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:08.637743 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffhnlm" event={"ID":"0fc24815-f5ae-4e3f-8cca-6637468415a2","Type":"ContainerDied","Data":"564f34874888e87412f66fb89ffa74e80f3727fd84642a6cdadda8ea76f92a3b"} Jul 08 19:09:08.637861 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:08.637794 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffhnlm" event={"ID":"0fc24815-f5ae-4e3f-8cca-6637468415a2","Type":"ContainerStarted","Data":"b8a3799caedbb3fe6074913affccfad4c71ccf5362799501dfd5070028860cf7"} Jul 08 19:09:08.927849 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:08.927776 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-kjczz"] Jul 08 19:09:08.930081 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:08.930062 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-kjczz" Jul 08 19:09:08.933606 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:08.932726 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-6jx44\"" Jul 08 19:09:08.933606 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:08.932726 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 08 19:09:08.939153 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:08.938829 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 08 19:09:08.939696 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:08.939672 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-kjczz"] Jul 08 19:09:09.023753 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:09.023715 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j97xh\" (UniqueName: \"kubernetes.io/projected/d5082b99-f831-4481-9aac-a267bbc1e3a0-kube-api-access-j97xh\") pod \"cert-manager-webhook-6486ffd997-kjczz\" (UID: \"d5082b99-f831-4481-9aac-a267bbc1e3a0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kjczz" Jul 08 19:09:09.023910 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:09.023767 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d5082b99-f831-4481-9aac-a267bbc1e3a0-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-kjczz\" (UID: \"d5082b99-f831-4481-9aac-a267bbc1e3a0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kjczz" Jul 08 19:09:09.124319 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:09.124284 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d5082b99-f831-4481-9aac-a267bbc1e3a0-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-kjczz\" (UID: \"d5082b99-f831-4481-9aac-a267bbc1e3a0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kjczz" Jul 08 19:09:09.124451 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:09.124349 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j97xh\" (UniqueName: \"kubernetes.io/projected/d5082b99-f831-4481-9aac-a267bbc1e3a0-kube-api-access-j97xh\") pod \"cert-manager-webhook-6486ffd997-kjczz\" (UID: \"d5082b99-f831-4481-9aac-a267bbc1e3a0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kjczz" Jul 08 19:09:09.133056 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:09.133018 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d5082b99-f831-4481-9aac-a267bbc1e3a0-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-kjczz\" (UID: \"d5082b99-f831-4481-9aac-a267bbc1e3a0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kjczz" Jul 08 19:09:09.133182 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:09.133056 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j97xh\" (UniqueName: \"kubernetes.io/projected/d5082b99-f831-4481-9aac-a267bbc1e3a0-kube-api-access-j97xh\") pod \"cert-manager-webhook-6486ffd997-kjczz\" (UID: \"d5082b99-f831-4481-9aac-a267bbc1e3a0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kjczz" Jul 08 19:09:09.258262 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:09.258181 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-kjczz" Jul 08 19:09:09.402834 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:09.402805 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-kjczz"] Jul 08 19:09:09.406069 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:09:09.406036 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5082b99_f831_4481_9aac_a267bbc1e3a0.slice/crio-5f71c55d6f1234576c8721b61c21e59f21618cbeab4643bff55fe9b39abe4a06 WatchSource:0}: Error finding container 5f71c55d6f1234576c8721b61c21e59f21618cbeab4643bff55fe9b39abe4a06: Status 404 returned error can't find the container with id 5f71c55d6f1234576c8721b61c21e59f21618cbeab4643bff55fe9b39abe4a06 Jul 08 19:09:09.644068 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:09.644026 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-kjczz" event={"ID":"d5082b99-f831-4481-9aac-a267bbc1e3a0","Type":"ContainerStarted","Data":"5f71c55d6f1234576c8721b61c21e59f21618cbeab4643bff55fe9b39abe4a06"} Jul 08 19:09:10.038357 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:10.038273 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:09:10.230324 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:10.230290 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:09:10.421784 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:10.421751 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-xxvls"] Jul 08 19:09:10.424288 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:10.424265 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xxvls" Jul 08 19:09:10.426792 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:10.426772 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-mlrrw\"" Jul 08 19:09:10.458041 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:10.457988 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-xxvls"] Jul 08 19:09:10.535476 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:10.535445 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhzdv\" (UniqueName: \"kubernetes.io/projected/2343e315-c7f5-4018-8021-ffd50b438fd5-kube-api-access-dhzdv\") pod \"cert-manager-cainjector-559cdd8c87-xxvls\" (UID: \"2343e315-c7f5-4018-8021-ffd50b438fd5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xxvls" Jul 08 19:09:10.535595 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:10.535506 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2343e315-c7f5-4018-8021-ffd50b438fd5-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-xxvls\" (UID: \"2343e315-c7f5-4018-8021-ffd50b438fd5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xxvls" Jul 08 19:09:10.636855 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:10.636829 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhzdv\" (UniqueName: \"kubernetes.io/projected/2343e315-c7f5-4018-8021-ffd50b438fd5-kube-api-access-dhzdv\") pod \"cert-manager-cainjector-559cdd8c87-xxvls\" (UID: \"2343e315-c7f5-4018-8021-ffd50b438fd5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xxvls" Jul 08 19:09:10.636992 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:10.636888 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2343e315-c7f5-4018-8021-ffd50b438fd5-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-xxvls\" (UID: \"2343e315-c7f5-4018-8021-ffd50b438fd5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xxvls" Jul 08 19:09:10.648676 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:10.648652 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2343e315-c7f5-4018-8021-ffd50b438fd5-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-xxvls\" (UID: \"2343e315-c7f5-4018-8021-ffd50b438fd5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xxvls" Jul 08 19:09:10.648892 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:10.648868 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhzdv\" (UniqueName: \"kubernetes.io/projected/2343e315-c7f5-4018-8021-ffd50b438fd5-kube-api-access-dhzdv\") pod \"cert-manager-cainjector-559cdd8c87-xxvls\" (UID: \"2343e315-c7f5-4018-8021-ffd50b438fd5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xxvls" Jul 08 19:09:10.651206 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:10.651145 2558 generic.go:356] "Generic (PLEG): container finished" podID="0fc24815-f5ae-4e3f-8cca-6637468415a2" containerID="40ef9ee108265b250ed151b2f8820daef4b466a72317fedbbf104556e8f47108" exitCode=0 Jul 08 19:09:10.651321 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:10.651244 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffhnlm" event={"ID":"0fc24815-f5ae-4e3f-8cca-6637468415a2","Type":"ContainerDied","Data":"40ef9ee108265b250ed151b2f8820daef4b466a72317fedbbf104556e8f47108"} Jul 08 19:09:10.733809 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:10.733779 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xxvls" Jul 08 19:09:10.883940 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:10.883893 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-xxvls"] Jul 08 19:09:10.887270 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:09:10.887240 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2343e315_c7f5_4018_8021_ffd50b438fd5.slice/crio-d8670a7a025da8014aa371b6ce744aedf3137ff242c0c27dd90c5fadf0e49287 WatchSource:0}: Error finding container d8670a7a025da8014aa371b6ce744aedf3137ff242c0c27dd90c5fadf0e49287: Status 404 returned error can't find the container with id d8670a7a025da8014aa371b6ce744aedf3137ff242c0c27dd90c5fadf0e49287 Jul 08 19:09:11.659617 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:11.659477 2558 generic.go:356] "Generic (PLEG): container finished" podID="0fc24815-f5ae-4e3f-8cca-6637468415a2" containerID="ed2567f5ce35f8f5eaa6a1b7242dfd396c29e63f4fa6b4ef06f747d22abe10ed" exitCode=0 Jul 08 19:09:11.659617 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:11.659525 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffhnlm" event={"ID":"0fc24815-f5ae-4e3f-8cca-6637468415a2","Type":"ContainerDied","Data":"ed2567f5ce35f8f5eaa6a1b7242dfd396c29e63f4fa6b4ef06f747d22abe10ed"} Jul 08 19:09:11.661204 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:11.661176 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xxvls" event={"ID":"2343e315-c7f5-4018-8021-ffd50b438fd5","Type":"ContainerStarted","Data":"d8670a7a025da8014aa371b6ce744aedf3137ff242c0c27dd90c5fadf0e49287"} Jul 08 19:09:12.692900 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:12.692872 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:09:12.793156 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:12.793130 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:09:12.810166 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:12.810141 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffhnlm" Jul 08 19:09:12.957874 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:12.957779 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbsrv\" (UniqueName: \"kubernetes.io/projected/0fc24815-f5ae-4e3f-8cca-6637468415a2-kube-api-access-hbsrv\") pod \"0fc24815-f5ae-4e3f-8cca-6637468415a2\" (UID: \"0fc24815-f5ae-4e3f-8cca-6637468415a2\") " Jul 08 19:09:12.957874 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:12.957849 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fc24815-f5ae-4e3f-8cca-6637468415a2-util\") pod \"0fc24815-f5ae-4e3f-8cca-6637468415a2\" (UID: \"0fc24815-f5ae-4e3f-8cca-6637468415a2\") " Jul 08 19:09:12.958093 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:12.957895 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fc24815-f5ae-4e3f-8cca-6637468415a2-bundle\") pod \"0fc24815-f5ae-4e3f-8cca-6637468415a2\" (UID: \"0fc24815-f5ae-4e3f-8cca-6637468415a2\") " Jul 08 19:09:12.958360 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:12.958335 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc24815-f5ae-4e3f-8cca-6637468415a2-bundle" (OuterVolumeSpecName: "bundle") pod "0fc24815-f5ae-4e3f-8cca-6637468415a2" (UID: "0fc24815-f5ae-4e3f-8cca-6637468415a2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:09:12.960208 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:12.960173 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fc24815-f5ae-4e3f-8cca-6637468415a2-kube-api-access-hbsrv" (OuterVolumeSpecName: "kube-api-access-hbsrv") pod "0fc24815-f5ae-4e3f-8cca-6637468415a2" (UID: "0fc24815-f5ae-4e3f-8cca-6637468415a2"). InnerVolumeSpecName "kube-api-access-hbsrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:09:12.966814 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:12.966785 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc24815-f5ae-4e3f-8cca-6637468415a2-util" (OuterVolumeSpecName: "util") pod "0fc24815-f5ae-4e3f-8cca-6637468415a2" (UID: "0fc24815-f5ae-4e3f-8cca-6637468415a2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:09:13.059110 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:13.059078 2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fc24815-f5ae-4e3f-8cca-6637468415a2-util\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:09:13.059110 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:13.059104 2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fc24815-f5ae-4e3f-8cca-6637468415a2-bundle\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:09:13.059110 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:13.059115 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbsrv\" (UniqueName: \"kubernetes.io/projected/0fc24815-f5ae-4e3f-8cca-6637468415a2-kube-api-access-hbsrv\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:09:13.672154 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:13.672121 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffhnlm" event={"ID":"0fc24815-f5ae-4e3f-8cca-6637468415a2","Type":"ContainerDied","Data":"b8a3799caedbb3fe6074913affccfad4c71ccf5362799501dfd5070028860cf7"} Jul 08 19:09:13.672341 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:13.672166 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8a3799caedbb3fe6074913affccfad4c71ccf5362799501dfd5070028860cf7" Jul 08 19:09:13.672341 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:13.672130 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffhnlm" Jul 08 19:09:13.673763 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:13.673737 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xxvls" event={"ID":"2343e315-c7f5-4018-8021-ffd50b438fd5","Type":"ContainerStarted","Data":"695f75b45c661cd10de7b27048c73a7ca295046b8f81506b7cad9e5c3eaf8d29"} Jul 08 19:09:13.675346 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:13.675314 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-kjczz" event={"ID":"d5082b99-f831-4481-9aac-a267bbc1e3a0","Type":"ContainerStarted","Data":"2371d2caf2777cfbb50a1b05f61968ba0943886935c3741e4a2ef41f6d35d9e8"} Jul 08 19:09:13.675450 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:13.675435 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-kjczz" Jul 08 19:09:13.692139 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:13.692095 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xxvls" podStartSLOduration=1.92675214 podStartE2EDuration="3.692083442s" totalImagesPullingTime="1.765331302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:09:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:09:13.691318857 +0000 UTC m=+336.950271802" watchObservedRunningTime="2026-07-08 19:09:13.692083442 +0000 UTC m=+336.951036366" Jul 08 19:09:13.707466 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:13.707423 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-kjczz" podStartSLOduration=2.467329437 podStartE2EDuration="5.707413072s" totalImagesPullingTime="3.240083635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:09:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:09:13.706429274 +0000 UTC m=+336.965382200" watchObservedRunningTime="2026-07-08 19:09:13.707413072 +0000 UTC m=+336.966365997" Jul 08 19:09:16.149685 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:16.149643 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:09:16.248543 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:16.248504 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:09:18.236472 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:18.236435 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:09:19.405054 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:19.405023 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:09:19.506066 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:19.506021 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:09:19.682288 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:19.682258 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-kjczz" Jul 08 19:09:24.363562 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:24.363530 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:09:24.469276 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:24.469243 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:09:25.978064 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:25.978027 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79l2h"] Jul 08 19:09:25.978408 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:25.978300 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fc24815-f5ae-4e3f-8cca-6637468415a2" containerName="util" Jul 08 19:09:25.978408 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:25.978311 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fc24815-f5ae-4e3f-8cca-6637468415a2" containerName="util" Jul 08 19:09:25.978408 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:25.978325 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fc24815-f5ae-4e3f-8cca-6637468415a2" containerName="pull" Jul 08 19:09:25.978408 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:25.978330 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fc24815-f5ae-4e3f-8cca-6637468415a2" containerName="pull" Jul 08 19:09:25.978408 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:25.978340 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fc24815-f5ae-4e3f-8cca-6637468415a2" containerName="extract" Jul 08 19:09:25.978408 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:25.978344 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fc24815-f5ae-4e3f-8cca-6637468415a2" containerName="extract" Jul 08 19:09:25.978408 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:25.978393 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fc24815-f5ae-4e3f-8cca-6637468415a2" containerName="extract" Jul 08 19:09:25.980470 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:25.980453 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79l2h" Jul 08 19:09:25.982977 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:25.982943 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 08 19:09:25.983113 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:25.983088 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 08 19:09:25.984006 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:25.983987 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lm5m4\"" Jul 08 19:09:25.989674 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:25.989655 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79l2h"] Jul 08 19:09:26.069420 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:26.069393 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3e3d573-63f4-4931-8d60-f2bf105963e3-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79l2h\" (UID: \"e3e3d573-63f4-4931-8d60-f2bf105963e3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79l2h" Jul 08 19:09:26.069530 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:26.069450 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3e3d573-63f4-4931-8d60-f2bf105963e3-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79l2h\" (UID: \"e3e3d573-63f4-4931-8d60-f2bf105963e3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79l2h" Jul 08 19:09:26.069530 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:26.069482 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6p9q\" (UniqueName: \"kubernetes.io/projected/e3e3d573-63f4-4931-8d60-f2bf105963e3-kube-api-access-d6p9q\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79l2h\" (UID: \"e3e3d573-63f4-4931-8d60-f2bf105963e3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79l2h" Jul 08 19:09:26.170773 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:26.170739 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3e3d573-63f4-4931-8d60-f2bf105963e3-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79l2h\" (UID: \"e3e3d573-63f4-4931-8d60-f2bf105963e3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79l2h" Jul 08 19:09:26.170950 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:26.170783 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6p9q\" (UniqueName: \"kubernetes.io/projected/e3e3d573-63f4-4931-8d60-f2bf105963e3-kube-api-access-d6p9q\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79l2h\" (UID: \"e3e3d573-63f4-4931-8d60-f2bf105963e3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79l2h" Jul 08 19:09:26.170950 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:26.170821 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3e3d573-63f4-4931-8d60-f2bf105963e3-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79l2h\" (UID: \"e3e3d573-63f4-4931-8d60-f2bf105963e3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79l2h" Jul 08 19:09:26.171116 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:26.171094 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3e3d573-63f4-4931-8d60-f2bf105963e3-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79l2h\" (UID: \"e3e3d573-63f4-4931-8d60-f2bf105963e3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79l2h" Jul 08 19:09:26.171162 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:26.171126 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3e3d573-63f4-4931-8d60-f2bf105963e3-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79l2h\" (UID: \"e3e3d573-63f4-4931-8d60-f2bf105963e3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79l2h" Jul 08 19:09:26.179078 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:26.179052 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6p9q\" (UniqueName: \"kubernetes.io/projected/e3e3d573-63f4-4931-8d60-f2bf105963e3-kube-api-access-d6p9q\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79l2h\" (UID: \"e3e3d573-63f4-4931-8d60-f2bf105963e3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79l2h" Jul 08 19:09:26.279061 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:26.278983 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:09:26.289973 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:26.289948 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79l2h" Jul 08 19:09:26.419875 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:26.419830 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79l2h"] Jul 08 19:09:26.421938 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:09:26.421909 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3e3d573_63f4_4931_8d60_f2bf105963e3.slice/crio-fe2fc3722ef64384cab09674658faff2e47d7a3183ccd8a6ccd2a0018b6a0278 WatchSource:0}: Error finding container fe2fc3722ef64384cab09674658faff2e47d7a3183ccd8a6ccd2a0018b6a0278: Status 404 returned error can't find the container with id fe2fc3722ef64384cab09674658faff2e47d7a3183ccd8a6ccd2a0018b6a0278 Jul 08 19:09:26.727997 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:26.727963 2558 generic.go:356] "Generic (PLEG): container finished" podID="e3e3d573-63f4-4931-8d60-f2bf105963e3" containerID="fcd45bd04040b2301707d9dd18b47ed3a7b22c274ba9402d1ba68b5f567b6da1" exitCode=0 Jul 08 19:09:26.728176 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:26.728007 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79l2h" event={"ID":"e3e3d573-63f4-4931-8d60-f2bf105963e3","Type":"ContainerDied","Data":"fcd45bd04040b2301707d9dd18b47ed3a7b22c274ba9402d1ba68b5f567b6da1"} Jul 08 19:09:26.728176 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:26.728042 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79l2h" event={"ID":"e3e3d573-63f4-4931-8d60-f2bf105963e3","Type":"ContainerStarted","Data":"fe2fc3722ef64384cab09674658faff2e47d7a3183ccd8a6ccd2a0018b6a0278"} Jul 08 19:09:27.733791 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:27.733756 2558 generic.go:356] "Generic (PLEG): container finished" podID="e3e3d573-63f4-4931-8d60-f2bf105963e3" containerID="22cc09add4ec99d5c3063b561b119abac30217d94d856b01b709b1bee596b094" exitCode=0 Jul 08 19:09:27.734162 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:27.733846 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79l2h" event={"ID":"e3e3d573-63f4-4931-8d60-f2bf105963e3","Type":"ContainerDied","Data":"22cc09add4ec99d5c3063b561b119abac30217d94d856b01b709b1bee596b094"} Jul 08 19:09:28.740945 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:28.740911 2558 generic.go:356] "Generic (PLEG): container finished" podID="e3e3d573-63f4-4931-8d60-f2bf105963e3" containerID="fc3fce7c13e0886d0ae5375ad4654ff2d6a855f6509ca4943bc591b304fa7f20" exitCode=0 Jul 08 19:09:28.741352 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:28.740995 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79l2h" event={"ID":"e3e3d573-63f4-4931-8d60-f2bf105963e3","Type":"ContainerDied","Data":"fc3fce7c13e0886d0ae5375ad4654ff2d6a855f6509ca4943bc591b304fa7f20"} Jul 08 19:09:29.873569 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:29.873543 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79l2h" Jul 08 19:09:30.003353 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:30.003278 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3e3d573-63f4-4931-8d60-f2bf105963e3-bundle\") pod \"e3e3d573-63f4-4931-8d60-f2bf105963e3\" (UID: \"e3e3d573-63f4-4931-8d60-f2bf105963e3\") " Jul 08 19:09:30.003353 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:30.003333 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3e3d573-63f4-4931-8d60-f2bf105963e3-util\") pod \"e3e3d573-63f4-4931-8d60-f2bf105963e3\" (UID: \"e3e3d573-63f4-4931-8d60-f2bf105963e3\") " Jul 08 19:09:30.003563 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:30.003377 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6p9q\" (UniqueName: \"kubernetes.io/projected/e3e3d573-63f4-4931-8d60-f2bf105963e3-kube-api-access-d6p9q\") pod \"e3e3d573-63f4-4931-8d60-f2bf105963e3\" (UID: \"e3e3d573-63f4-4931-8d60-f2bf105963e3\") " Jul 08 19:09:30.004009 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:30.003980 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3e3d573-63f4-4931-8d60-f2bf105963e3-bundle" (OuterVolumeSpecName: "bundle") pod "e3e3d573-63f4-4931-8d60-f2bf105963e3" (UID: "e3e3d573-63f4-4931-8d60-f2bf105963e3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:09:30.005444 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:30.005418 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3e3d573-63f4-4931-8d60-f2bf105963e3-kube-api-access-d6p9q" (OuterVolumeSpecName: "kube-api-access-d6p9q") pod "e3e3d573-63f4-4931-8d60-f2bf105963e3" (UID: "e3e3d573-63f4-4931-8d60-f2bf105963e3"). InnerVolumeSpecName "kube-api-access-d6p9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:09:30.009219 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:30.009178 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3e3d573-63f4-4931-8d60-f2bf105963e3-util" (OuterVolumeSpecName: "util") pod "e3e3d573-63f4-4931-8d60-f2bf105963e3" (UID: "e3e3d573-63f4-4931-8d60-f2bf105963e3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:09:30.104204 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:30.104175 2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3e3d573-63f4-4931-8d60-f2bf105963e3-bundle\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:09:30.104204 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:30.104199 2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3e3d573-63f4-4931-8d60-f2bf105963e3-util\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:09:30.104340 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:30.104210 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6p9q\" (UniqueName: \"kubernetes.io/projected/e3e3d573-63f4-4931-8d60-f2bf105963e3-kube-api-access-d6p9q\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:09:30.752480 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:30.752451 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79l2h" Jul 08 19:09:30.752709 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:30.752450 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af79l2h" event={"ID":"e3e3d573-63f4-4931-8d60-f2bf105963e3","Type":"ContainerDied","Data":"fe2fc3722ef64384cab09674658faff2e47d7a3183ccd8a6ccd2a0018b6a0278"} Jul 08 19:09:30.752709 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:30.752573 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe2fc3722ef64384cab09674658faff2e47d7a3183ccd8a6ccd2a0018b6a0278" Jul 08 19:09:39.946129 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:39.946081 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:09:40.036965 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:40.036925 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:09:40.751596 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:40.751540 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:09:41.220239 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:41.220204 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:09:41.330421 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:41.330379 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:09:41.384677 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:41.384636 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-994cc9f69-2t2nf"] Jul 08 19:09:41.384958 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:41.384946 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3e3d573-63f4-4931-8d60-f2bf105963e3" containerName="pull" Jul 08 19:09:41.385000 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:41.384960 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3e3d573-63f4-4931-8d60-f2bf105963e3" containerName="pull" Jul 08 19:09:41.385000 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:41.384973 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3e3d573-63f4-4931-8d60-f2bf105963e3" containerName="extract" Jul 08 19:09:41.385000 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:41.384979 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3e3d573-63f4-4931-8d60-f2bf105963e3" containerName="extract" Jul 08 19:09:41.385000 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:41.384991 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3e3d573-63f4-4931-8d60-f2bf105963e3" containerName="util" Jul 08 19:09:41.385000 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:41.384996 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3e3d573-63f4-4931-8d60-f2bf105963e3" containerName="util" Jul 08 19:09:41.385179 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:41.385063 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3e3d573-63f4-4931-8d60-f2bf105963e3" containerName="extract" Jul 08 19:09:41.389273 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:41.389253 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-2t2nf" Jul 08 19:09:41.394626 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:41.394608 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 08 19:09:41.396429 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:41.396364 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 08 19:09:41.396429 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:41.396392 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 08 19:09:41.397538 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:41.397523 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 08 19:09:41.398220 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:41.398205 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-dkxt2\"" Jul 08 19:09:41.436893 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:41.436870 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-994cc9f69-2t2nf"] Jul 08 19:09:41.488774 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:41.488702 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/02e501c0-f92d-4d99-946e-2d2344c1dd99-webhook-cert\") pod \"opendatahub-operator-controller-manager-994cc9f69-2t2nf\" (UID: \"02e501c0-f92d-4d99-946e-2d2344c1dd99\") " pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-2t2nf" Jul 08 19:09:41.488774 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:41.488741 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02e501c0-f92d-4d99-946e-2d2344c1dd99-tmp\") pod \"opendatahub-operator-controller-manager-994cc9f69-2t2nf\" (UID: \"02e501c0-f92d-4d99-946e-2d2344c1dd99\") " pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-2t2nf" Jul 08 19:09:41.488938 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:41.488778 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/02e501c0-f92d-4d99-946e-2d2344c1dd99-manifests\") pod \"opendatahub-operator-controller-manager-994cc9f69-2t2nf\" (UID: \"02e501c0-f92d-4d99-946e-2d2344c1dd99\") " pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-2t2nf" Jul 08 19:09:41.488938 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:41.488802 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/02e501c0-f92d-4d99-946e-2d2344c1dd99-apiservice-cert\") pod \"opendatahub-operator-controller-manager-994cc9f69-2t2nf\" (UID: \"02e501c0-f92d-4d99-946e-2d2344c1dd99\") " pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-2t2nf" Jul 08 19:09:41.488938 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:41.488829 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llgxg\" (UniqueName: \"kubernetes.io/projected/02e501c0-f92d-4d99-946e-2d2344c1dd99-kube-api-access-llgxg\") pod \"opendatahub-operator-controller-manager-994cc9f69-2t2nf\" (UID: \"02e501c0-f92d-4d99-946e-2d2344c1dd99\") " pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-2t2nf" Jul 08 19:09:41.589274 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:41.589240 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/02e501c0-f92d-4d99-946e-2d2344c1dd99-manifests\") pod \"opendatahub-operator-controller-manager-994cc9f69-2t2nf\" (UID: \"02e501c0-f92d-4d99-946e-2d2344c1dd99\") " pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-2t2nf" Jul 08 19:09:41.589274 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:41.589278 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/02e501c0-f92d-4d99-946e-2d2344c1dd99-apiservice-cert\") pod \"opendatahub-operator-controller-manager-994cc9f69-2t2nf\" (UID: \"02e501c0-f92d-4d99-946e-2d2344c1dd99\") " pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-2t2nf" Jul 08 19:09:41.589493 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:41.589319 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llgxg\" (UniqueName: \"kubernetes.io/projected/02e501c0-f92d-4d99-946e-2d2344c1dd99-kube-api-access-llgxg\") pod \"opendatahub-operator-controller-manager-994cc9f69-2t2nf\" (UID: \"02e501c0-f92d-4d99-946e-2d2344c1dd99\") " pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-2t2nf" Jul 08 19:09:41.589493 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:41.589367 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/02e501c0-f92d-4d99-946e-2d2344c1dd99-webhook-cert\") pod \"opendatahub-operator-controller-manager-994cc9f69-2t2nf\" (UID: \"02e501c0-f92d-4d99-946e-2d2344c1dd99\") " pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-2t2nf" Jul 08 19:09:41.589493 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:41.589411 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02e501c0-f92d-4d99-946e-2d2344c1dd99-tmp\") pod \"opendatahub-operator-controller-manager-994cc9f69-2t2nf\" (UID: \"02e501c0-f92d-4d99-946e-2d2344c1dd99\") " pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-2t2nf" Jul 08 19:09:41.589725 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:41.589703 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/02e501c0-f92d-4d99-946e-2d2344c1dd99-manifests\") pod \"opendatahub-operator-controller-manager-994cc9f69-2t2nf\" (UID: \"02e501c0-f92d-4d99-946e-2d2344c1dd99\") " pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-2t2nf" Jul 08 19:09:41.591535 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:41.591507 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02e501c0-f92d-4d99-946e-2d2344c1dd99-tmp\") pod \"opendatahub-operator-controller-manager-994cc9f69-2t2nf\" (UID: \"02e501c0-f92d-4d99-946e-2d2344c1dd99\") " pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-2t2nf" Jul 08 19:09:41.591798 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:41.591781 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/02e501c0-f92d-4d99-946e-2d2344c1dd99-apiservice-cert\") pod \"opendatahub-operator-controller-manager-994cc9f69-2t2nf\" (UID: \"02e501c0-f92d-4d99-946e-2d2344c1dd99\") " pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-2t2nf" Jul 08 19:09:41.591863 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:41.591845 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/02e501c0-f92d-4d99-946e-2d2344c1dd99-webhook-cert\") pod \"opendatahub-operator-controller-manager-994cc9f69-2t2nf\" (UID: \"02e501c0-f92d-4d99-946e-2d2344c1dd99\") " pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-2t2nf" Jul 08 19:09:41.598381 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:41.598337 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:09:41.600897 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:41.600876 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llgxg\" (UniqueName: \"kubernetes.io/projected/02e501c0-f92d-4d99-946e-2d2344c1dd99-kube-api-access-llgxg\") pod \"opendatahub-operator-controller-manager-994cc9f69-2t2nf\" (UID: \"02e501c0-f92d-4d99-946e-2d2344c1dd99\") " pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-2t2nf" Jul 08 19:09:41.699025 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:41.698997 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-2t2nf" Jul 08 19:09:41.716085 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:41.716056 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:09:41.814186 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:41.814140 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:09:41.836850 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:41.836822 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-994cc9f69-2t2nf"] Jul 08 19:09:41.840493 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:09:41.840472 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02e501c0_f92d_4d99_946e_2d2344c1dd99.slice/crio-fedb361ed3ce11a670de41c96c674403579cc5d1815f1161851ae45a1f104388 WatchSource:0}: Error finding container fedb361ed3ce11a670de41c96c674403579cc5d1815f1161851ae45a1f104388: Status 404 returned error can't find the container with id fedb361ed3ce11a670de41c96c674403579cc5d1815f1161851ae45a1f104388 Jul 08 19:09:41.910378 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:41.910353 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:09:42.001057 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:42.000988 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:09:42.091884 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:42.091833 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:09:42.187280 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:42.187242 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:09:42.291097 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:42.291017 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:09:42.806205 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:42.806156 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-2t2nf" event={"ID":"02e501c0-f92d-4d99-946e-2d2344c1dd99","Type":"ContainerStarted","Data":"fedb361ed3ce11a670de41c96c674403579cc5d1815f1161851ae45a1f104388"} Jul 08 19:09:44.816823 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:44.816785 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-2t2nf" event={"ID":"02e501c0-f92d-4d99-946e-2d2344c1dd99","Type":"ContainerStarted","Data":"e00eb15cee57bfd8d1be4e0e6973e36459873e80c21763c8ad4f82337405de15"} Jul 08 19:09:45.823048 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:45.823015 2558 generic.go:356] "Generic (PLEG): container finished" podID="02e501c0-f92d-4d99-946e-2d2344c1dd99" containerID="e00eb15cee57bfd8d1be4e0e6973e36459873e80c21763c8ad4f82337405de15" exitCode=0 Jul 08 19:09:45.823502 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:45.823083 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-2t2nf" event={"ID":"02e501c0-f92d-4d99-946e-2d2344c1dd99","Type":"ContainerDied","Data":"e00eb15cee57bfd8d1be4e0e6973e36459873e80c21763c8ad4f82337405de15"} Jul 08 19:09:46.160459 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:46.160428 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:09:46.261661 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:46.261614 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:09:46.830878 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:46.830842 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-2t2nf" event={"ID":"02e501c0-f92d-4d99-946e-2d2344c1dd99","Type":"ContainerStarted","Data":"10a722e076fd8bcd2d830148c9875954ca980c8e0ecbe7a5058d5acf59f56887"} Jul 08 19:09:46.831340 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:46.830988 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-2t2nf" Jul 08 19:09:46.862793 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:46.862739 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-2t2nf" podStartSLOduration=1.521122106 podStartE2EDuration="5.862718771s" totalImagesPullingTime="3.254122489s" totalInitContainerRuntime="1.087474176s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:09:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:09:46.860017043 +0000 UTC m=+370.118969993" watchObservedRunningTime="2026-07-08 19:09:46.862718771 +0000 UTC m=+370.121671697" Jul 08 19:09:52.218057 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:52.218017 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:09:52.314967 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:52.314931 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:09:54.536285 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:54.536252 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:09:54.642680 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:54.642647 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:09:57.838272 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:57.838240 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-2t2nf" Jul 08 19:09:58.817559 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:58.817524 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-69dfbfd578-2h524"] Jul 08 19:09:58.823764 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:58.823738 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-69dfbfd578-2h524" Jul 08 19:09:58.827251 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:58.827222 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 08 19:09:58.827251 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:58.827243 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 08 19:09:58.827420 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:58.827244 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 08 19:09:58.827516 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:58.827500 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-v7zqq\"" Jul 08 19:09:58.827608 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:58.827524 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 08 19:09:58.827608 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:58.827563 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 08 19:09:58.832008 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:58.831989 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-69dfbfd578-2h524"] Jul 08 19:09:58.917939 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:58.917914 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f1b3c05-4c06-4f57-a739-20f77855c0ee-cert\") pod \"lws-controller-manager-69dfbfd578-2h524\" (UID: \"6f1b3c05-4c06-4f57-a739-20f77855c0ee\") " pod="openshift-lws-operator/lws-controller-manager-69dfbfd578-2h524" Jul 08 19:09:58.918293 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:58.917946 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t7wk\" (UniqueName: \"kubernetes.io/projected/6f1b3c05-4c06-4f57-a739-20f77855c0ee-kube-api-access-6t7wk\") pod \"lws-controller-manager-69dfbfd578-2h524\" (UID: \"6f1b3c05-4c06-4f57-a739-20f77855c0ee\") " pod="openshift-lws-operator/lws-controller-manager-69dfbfd578-2h524" Jul 08 19:09:58.918293 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:58.917992 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f1b3c05-4c06-4f57-a739-20f77855c0ee-metrics-cert\") pod \"lws-controller-manager-69dfbfd578-2h524\" (UID: \"6f1b3c05-4c06-4f57-a739-20f77855c0ee\") " pod="openshift-lws-operator/lws-controller-manager-69dfbfd578-2h524" Jul 08 19:09:58.918293 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:58.918026 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6f1b3c05-4c06-4f57-a739-20f77855c0ee-manager-config\") pod \"lws-controller-manager-69dfbfd578-2h524\" (UID: \"6f1b3c05-4c06-4f57-a739-20f77855c0ee\") " pod="openshift-lws-operator/lws-controller-manager-69dfbfd578-2h524" Jul 08 19:09:59.019098 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:59.019063 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f1b3c05-4c06-4f57-a739-20f77855c0ee-metrics-cert\") pod \"lws-controller-manager-69dfbfd578-2h524\" (UID: \"6f1b3c05-4c06-4f57-a739-20f77855c0ee\") " pod="openshift-lws-operator/lws-controller-manager-69dfbfd578-2h524" Jul 08 19:09:59.019264 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:59.019107 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6f1b3c05-4c06-4f57-a739-20f77855c0ee-manager-config\") pod \"lws-controller-manager-69dfbfd578-2h524\" (UID: \"6f1b3c05-4c06-4f57-a739-20f77855c0ee\") " pod="openshift-lws-operator/lws-controller-manager-69dfbfd578-2h524" Jul 08 19:09:59.019264 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:59.019155 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f1b3c05-4c06-4f57-a739-20f77855c0ee-cert\") pod \"lws-controller-manager-69dfbfd578-2h524\" (UID: \"6f1b3c05-4c06-4f57-a739-20f77855c0ee\") " pod="openshift-lws-operator/lws-controller-manager-69dfbfd578-2h524" Jul 08 19:09:59.019264 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:59.019176 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t7wk\" (UniqueName: \"kubernetes.io/projected/6f1b3c05-4c06-4f57-a739-20f77855c0ee-kube-api-access-6t7wk\") pod \"lws-controller-manager-69dfbfd578-2h524\" (UID: \"6f1b3c05-4c06-4f57-a739-20f77855c0ee\") " pod="openshift-lws-operator/lws-controller-manager-69dfbfd578-2h524" Jul 08 19:09:59.019948 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:59.019922 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6f1b3c05-4c06-4f57-a739-20f77855c0ee-manager-config\") pod \"lws-controller-manager-69dfbfd578-2h524\" (UID: \"6f1b3c05-4c06-4f57-a739-20f77855c0ee\") " pod="openshift-lws-operator/lws-controller-manager-69dfbfd578-2h524" Jul 08 19:09:59.021610 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:59.021590 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f1b3c05-4c06-4f57-a739-20f77855c0ee-cert\") pod \"lws-controller-manager-69dfbfd578-2h524\" (UID: \"6f1b3c05-4c06-4f57-a739-20f77855c0ee\") " pod="openshift-lws-operator/lws-controller-manager-69dfbfd578-2h524" Jul 08 19:09:59.021699 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:59.021625 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f1b3c05-4c06-4f57-a739-20f77855c0ee-metrics-cert\") pod \"lws-controller-manager-69dfbfd578-2h524\" (UID: \"6f1b3c05-4c06-4f57-a739-20f77855c0ee\") " pod="openshift-lws-operator/lws-controller-manager-69dfbfd578-2h524" Jul 08 19:09:59.029880 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:59.029854 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t7wk\" (UniqueName: \"kubernetes.io/projected/6f1b3c05-4c06-4f57-a739-20f77855c0ee-kube-api-access-6t7wk\") pod \"lws-controller-manager-69dfbfd578-2h524\" (UID: \"6f1b3c05-4c06-4f57-a739-20f77855c0ee\") " pod="openshift-lws-operator/lws-controller-manager-69dfbfd578-2h524" Jul 08 19:09:59.133253 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:59.133221 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-69dfbfd578-2h524" Jul 08 19:09:59.287310 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:59.287227 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-69dfbfd578-2h524"] Jul 08 19:09:59.290021 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:09:59.289996 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f1b3c05_4c06_4f57_a739_20f77855c0ee.slice/crio-6f397644f415c342b4a230f999a1396a0020cc9e588e730aac3052c8cd37206c WatchSource:0}: Error finding container 6f397644f415c342b4a230f999a1396a0020cc9e588e730aac3052c8cd37206c: Status 404 returned error can't find the container with id 6f397644f415c342b4a230f999a1396a0020cc9e588e730aac3052c8cd37206c Jul 08 19:09:59.892563 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:09:59.892521 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-69dfbfd578-2h524" event={"ID":"6f1b3c05-4c06-4f57-a739-20f77855c0ee","Type":"ContainerStarted","Data":"6f397644f415c342b4a230f999a1396a0020cc9e588e730aac3052c8cd37206c"} Jul 08 19:10:00.422350 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:00.422317 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:10:00.537626 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:00.537598 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:10:00.899796 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:00.899761 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-69dfbfd578-2h524" event={"ID":"6f1b3c05-4c06-4f57-a739-20f77855c0ee","Type":"ContainerStarted","Data":"14274f1612ee6cb2e7160d07fec506ca51465b8fe62fa6d51dc8d437ed926a6e"} Jul 08 19:10:00.899968 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:00.899884 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-69dfbfd578-2h524" Jul 08 19:10:00.997147 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:00.997107 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:10:01.268918 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:01.268837 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:10:01.389785 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:01.389749 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:10:01.493398 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:01.493367 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:10:01.597955 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:01.597868 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:10:02.231505 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:02.231444 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:10:02.442366 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:02.442329 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:10:02.638333 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:02.638271 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:10:02.713621 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:02.713570 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:10:04.106546 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:04.106503 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:10:08.402870 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:08.402832 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:10:08.489146 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:08.489108 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:10:08.613956 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:08.613897 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-69dfbfd578-2h524" podStartSLOduration=9.461864804 podStartE2EDuration="10.613880436s" totalImagesPullingTime="1.152015632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:09:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:10:00.974893655 +0000 UTC m=+384.233846600" watchObservedRunningTime="2026-07-08 19:10:08.613880436 +0000 UTC m=+391.872833363" Jul 08 19:10:08.615358 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:08.615333 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s2rml"] Jul 08 19:10:08.618603 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:08.618568 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s2rml" Jul 08 19:10:08.633462 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:08.633432 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-5lfgg\"" Jul 08 19:10:08.633611 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:08.633468 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 08 19:10:08.633611 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:08.633478 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 08 19:10:08.638087 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:08.638065 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 08 19:10:08.641072 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:08.641051 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 08 19:10:08.684926 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:08.684858 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s2rml"] Jul 08 19:10:08.703261 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:08.703234 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/36f6816d-0f61-47ab-a861-fffd324ea5cf-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-s2rml\" (UID: \"36f6816d-0f61-47ab-a861-fffd324ea5cf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s2rml" Jul 08 19:10:08.703413 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:08.703267 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/36f6816d-0f61-47ab-a861-fffd324ea5cf-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-s2rml\" (UID: \"36f6816d-0f61-47ab-a861-fffd324ea5cf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s2rml" Jul 08 19:10:08.703413 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:08.703290 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/36f6816d-0f61-47ab-a861-fffd324ea5cf-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-s2rml\" (UID: \"36f6816d-0f61-47ab-a861-fffd324ea5cf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s2rml" Jul 08 19:10:08.703413 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:08.703346 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/36f6816d-0f61-47ab-a861-fffd324ea5cf-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-s2rml\" (UID: \"36f6816d-0f61-47ab-a861-fffd324ea5cf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s2rml" Jul 08 19:10:08.703413 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:08.703406 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts646\" (UniqueName: \"kubernetes.io/projected/36f6816d-0f61-47ab-a861-fffd324ea5cf-kube-api-access-ts646\") pod \"istiod-openshift-gateway-94bb8fbfd-s2rml\" (UID: \"36f6816d-0f61-47ab-a861-fffd324ea5cf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s2rml" Jul 08 19:10:08.703627 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:08.703454 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36f6816d-0f61-47ab-a861-fffd324ea5cf-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-s2rml\" (UID: \"36f6816d-0f61-47ab-a861-fffd324ea5cf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s2rml" Jul 08 19:10:08.703627 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:08.703547 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/36f6816d-0f61-47ab-a861-fffd324ea5cf-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-s2rml\" (UID: \"36f6816d-0f61-47ab-a861-fffd324ea5cf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s2rml" Jul 08 19:10:08.804142 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:08.804105 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36f6816d-0f61-47ab-a861-fffd324ea5cf-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-s2rml\" (UID: \"36f6816d-0f61-47ab-a861-fffd324ea5cf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s2rml" Jul 08 19:10:08.804142 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:08.804147 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/36f6816d-0f61-47ab-a861-fffd324ea5cf-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-s2rml\" (UID: \"36f6816d-0f61-47ab-a861-fffd324ea5cf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s2rml" Jul 08 19:10:08.804350 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:08.804191 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/36f6816d-0f61-47ab-a861-fffd324ea5cf-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-s2rml\" (UID: \"36f6816d-0f61-47ab-a861-fffd324ea5cf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s2rml" Jul 08 19:10:08.804350 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:08.804214 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/36f6816d-0f61-47ab-a861-fffd324ea5cf-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-s2rml\" (UID: \"36f6816d-0f61-47ab-a861-fffd324ea5cf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s2rml" Jul 08 19:10:08.804350 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:08.804235 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/36f6816d-0f61-47ab-a861-fffd324ea5cf-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-s2rml\" (UID: \"36f6816d-0f61-47ab-a861-fffd324ea5cf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s2rml" Jul 08 19:10:08.804350 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:08.804268 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/36f6816d-0f61-47ab-a861-fffd324ea5cf-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-s2rml\" (UID: \"36f6816d-0f61-47ab-a861-fffd324ea5cf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s2rml" Jul 08 19:10:08.804350 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:08.804310 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts646\" (UniqueName: \"kubernetes.io/projected/36f6816d-0f61-47ab-a861-fffd324ea5cf-kube-api-access-ts646\") pod \"istiod-openshift-gateway-94bb8fbfd-s2rml\" (UID: \"36f6816d-0f61-47ab-a861-fffd324ea5cf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s2rml" Jul 08 19:10:08.804994 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:08.804958 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/36f6816d-0f61-47ab-a861-fffd324ea5cf-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-s2rml\" (UID: \"36f6816d-0f61-47ab-a861-fffd324ea5cf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s2rml" Jul 08 19:10:08.806523 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:08.806498 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/36f6816d-0f61-47ab-a861-fffd324ea5cf-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-s2rml\" (UID: \"36f6816d-0f61-47ab-a861-fffd324ea5cf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s2rml" Jul 08 19:10:08.806709 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:08.806691 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/36f6816d-0f61-47ab-a861-fffd324ea5cf-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-s2rml\" (UID: \"36f6816d-0f61-47ab-a861-fffd324ea5cf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s2rml" Jul 08 19:10:08.806780 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:08.806700 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/36f6816d-0f61-47ab-a861-fffd324ea5cf-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-s2rml\" (UID: \"36f6816d-0f61-47ab-a861-fffd324ea5cf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s2rml" Jul 08 19:10:08.806780 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:08.806743 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/36f6816d-0f61-47ab-a861-fffd324ea5cf-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-s2rml\" (UID: \"36f6816d-0f61-47ab-a861-fffd324ea5cf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s2rml" Jul 08 19:10:08.830450 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:08.830416 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts646\" (UniqueName: \"kubernetes.io/projected/36f6816d-0f61-47ab-a861-fffd324ea5cf-kube-api-access-ts646\") pod \"istiod-openshift-gateway-94bb8fbfd-s2rml\" (UID: \"36f6816d-0f61-47ab-a861-fffd324ea5cf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s2rml" Jul 08 19:10:08.830571 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:08.830553 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36f6816d-0f61-47ab-a861-fffd324ea5cf-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-s2rml\" (UID: \"36f6816d-0f61-47ab-a861-fffd324ea5cf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s2rml" Jul 08 19:10:08.927814 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:08.927782 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s2rml" Jul 08 19:10:09.068728 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:09.068696 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s2rml"] Jul 08 19:10:09.070397 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:10:09.070370 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36f6816d_0f61_47ab_a861_fffd324ea5cf.slice/crio-b4438f92c85619cd4d12ad5824444c1ec85867ba4bf758033506034809484009 WatchSource:0}: Error finding container b4438f92c85619cd4d12ad5824444c1ec85867ba4bf758033506034809484009: Status 404 returned error can't find the container with id b4438f92c85619cd4d12ad5824444c1ec85867ba4bf758033506034809484009 Jul 08 19:10:09.392527 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:09.392490 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:10:09.487970 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:09.487939 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:10:09.623913 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:09.623869 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:10:09.955037 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:09.955003 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s2rml" event={"ID":"36f6816d-0f61-47ab-a861-fffd324ea5cf","Type":"ContainerStarted","Data":"b4438f92c85619cd4d12ad5824444c1ec85867ba4bf758033506034809484009"} Jul 08 19:10:11.515750 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:11.515706 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 08 19:10:11.516035 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:11.515778 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 08 19:10:11.906118 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:11.906094 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-69dfbfd578-2h524" Jul 08 19:10:11.967429 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:11.967391 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s2rml" event={"ID":"36f6816d-0f61-47ab-a861-fffd324ea5cf","Type":"ContainerStarted","Data":"7a240a26819c2014a1843e5b2d5c8b6553ba4a53024cb052f30f9f794ae7f17e"} Jul 08 19:10:11.967640 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:11.967487 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s2rml" Jul 08 19:10:12.491367 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:12.491310 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:10:12.630640 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:12.630610 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:10:12.728620 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:12.728562 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:10:12.885054 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:12.885014 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:10:12.973365 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:12.973341 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s2rml" Jul 08 19:10:13.029839 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:13.029805 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:10:13.051622 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:13.051558 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s2rml" podStartSLOduration=2.608410323 podStartE2EDuration="5.051541938s" totalImagesPullingTime="2.443131615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:10:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:10:12.10193437 +0000 UTC m=+395.360887297" watchObservedRunningTime="2026-07-08 19:10:13.051541938 +0000 UTC m=+396.310494863" Jul 08 19:10:13.144044 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:13.143944 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:10:13.251602 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:13.251549 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:10:13.348376 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:13.348347 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:10:46.041993 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:46.041951 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:10:46.094913 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:10:46.094877 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:11:01.976382 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:01.976289 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:11:02.082852 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:02.082810 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:11:02.230252 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:02.230168 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:11:02.934884 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:02.934841 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:11:03.131130 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:03.131086 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:11:17.742141 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:17.742106 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lqtlg"] Jul 08 19:11:17.744625 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:17.744606 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lqtlg" Jul 08 19:11:17.747071 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:17.747043 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 08 19:11:17.747177 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:17.747109 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 08 19:11:17.748040 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:17.748014 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2vkt6\"" Jul 08 19:11:17.756199 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:17.756169 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lqtlg"] Jul 08 19:11:17.878240 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:17.878209 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c24b2a5d-6bf4-4d86-a669-bcb498b921be-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lqtlg\" (UID: \"c24b2a5d-6bf4-4d86-a669-bcb498b921be\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lqtlg" Jul 08 19:11:17.878418 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:17.878253 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s52tj\" (UniqueName: \"kubernetes.io/projected/c24b2a5d-6bf4-4d86-a669-bcb498b921be-kube-api-access-s52tj\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lqtlg\" (UID: \"c24b2a5d-6bf4-4d86-a669-bcb498b921be\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lqtlg" Jul 08 19:11:17.878418 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:17.878293 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c24b2a5d-6bf4-4d86-a669-bcb498b921be-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lqtlg\" (UID: \"c24b2a5d-6bf4-4d86-a669-bcb498b921be\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lqtlg" Jul 08 19:11:17.979646 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:17.979605 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c24b2a5d-6bf4-4d86-a669-bcb498b921be-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lqtlg\" (UID: \"c24b2a5d-6bf4-4d86-a669-bcb498b921be\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lqtlg" Jul 08 19:11:17.979805 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:17.979699 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c24b2a5d-6bf4-4d86-a669-bcb498b921be-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lqtlg\" (UID: \"c24b2a5d-6bf4-4d86-a669-bcb498b921be\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lqtlg" Jul 08 19:11:17.979805 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:17.979743 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s52tj\" (UniqueName: \"kubernetes.io/projected/c24b2a5d-6bf4-4d86-a669-bcb498b921be-kube-api-access-s52tj\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lqtlg\" (UID: \"c24b2a5d-6bf4-4d86-a669-bcb498b921be\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lqtlg" Jul 08 19:11:17.980089 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:17.980069 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c24b2a5d-6bf4-4d86-a669-bcb498b921be-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lqtlg\" (UID: \"c24b2a5d-6bf4-4d86-a669-bcb498b921be\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lqtlg" Jul 08 19:11:17.980125 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:17.980108 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c24b2a5d-6bf4-4d86-a669-bcb498b921be-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lqtlg\" (UID: \"c24b2a5d-6bf4-4d86-a669-bcb498b921be\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lqtlg" Jul 08 19:11:17.989024 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:17.988988 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s52tj\" (UniqueName: \"kubernetes.io/projected/c24b2a5d-6bf4-4d86-a669-bcb498b921be-kube-api-access-s52tj\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lqtlg\" (UID: \"c24b2a5d-6bf4-4d86-a669-bcb498b921be\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lqtlg" Jul 08 19:11:18.053890 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:18.053815 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lqtlg" Jul 08 19:11:18.187425 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:18.187398 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lqtlg"] Jul 08 19:11:18.189388 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:11:18.189361 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc24b2a5d_6bf4_4d86_a669_bcb498b921be.slice/crio-490804b006159139a1cafc5b754b1650c794a349c91ecf65a0ee4c6b82c8f162 WatchSource:0}: Error finding container 490804b006159139a1cafc5b754b1650c794a349c91ecf65a0ee4c6b82c8f162: Status 404 returned error can't find the container with id 490804b006159139a1cafc5b754b1650c794a349c91ecf65a0ee4c6b82c8f162 Jul 08 19:11:18.274333 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:18.274299 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lqtlg" event={"ID":"c24b2a5d-6bf4-4d86-a669-bcb498b921be","Type":"ContainerStarted","Data":"4e25b2d56e852f86106f8d89790527aa35cf5cc8bfaab28e77028c11c3bf9fdd"} Jul 08 19:11:18.274495 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:18.274355 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lqtlg" event={"ID":"c24b2a5d-6bf4-4d86-a669-bcb498b921be","Type":"ContainerStarted","Data":"490804b006159139a1cafc5b754b1650c794a349c91ecf65a0ee4c6b82c8f162"} Jul 08 19:11:18.349684 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:18.349619 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mgrpx"] Jul 08 19:11:18.352044 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:18.352026 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mgrpx" Jul 08 19:11:18.363758 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:18.363730 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mgrpx"] Jul 08 19:11:18.484241 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:18.484211 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb40b1b2-9e96-46d1-9e8e-f0586f3938b7-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mgrpx\" (UID: \"bb40b1b2-9e96-46d1-9e8e-f0586f3938b7\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mgrpx" Jul 08 19:11:18.484413 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:18.484252 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb40b1b2-9e96-46d1-9e8e-f0586f3938b7-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mgrpx\" (UID: \"bb40b1b2-9e96-46d1-9e8e-f0586f3938b7\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mgrpx" Jul 08 19:11:18.484413 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:18.484319 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssdfp\" (UniqueName: \"kubernetes.io/projected/bb40b1b2-9e96-46d1-9e8e-f0586f3938b7-kube-api-access-ssdfp\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mgrpx\" (UID: \"bb40b1b2-9e96-46d1-9e8e-f0586f3938b7\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mgrpx" Jul 08 19:11:18.585385 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:18.585353 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssdfp\" (UniqueName: \"kubernetes.io/projected/bb40b1b2-9e96-46d1-9e8e-f0586f3938b7-kube-api-access-ssdfp\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mgrpx\" (UID: \"bb40b1b2-9e96-46d1-9e8e-f0586f3938b7\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mgrpx" Jul 08 19:11:18.585563 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:18.585437 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb40b1b2-9e96-46d1-9e8e-f0586f3938b7-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mgrpx\" (UID: \"bb40b1b2-9e96-46d1-9e8e-f0586f3938b7\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mgrpx" Jul 08 19:11:18.585563 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:18.585457 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb40b1b2-9e96-46d1-9e8e-f0586f3938b7-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mgrpx\" (UID: \"bb40b1b2-9e96-46d1-9e8e-f0586f3938b7\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mgrpx" Jul 08 19:11:18.585860 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:18.585843 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb40b1b2-9e96-46d1-9e8e-f0586f3938b7-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mgrpx\" (UID: \"bb40b1b2-9e96-46d1-9e8e-f0586f3938b7\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mgrpx" Jul 08 19:11:18.585914 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:18.585877 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb40b1b2-9e96-46d1-9e8e-f0586f3938b7-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mgrpx\" (UID: \"bb40b1b2-9e96-46d1-9e8e-f0586f3938b7\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mgrpx" Jul 08 19:11:18.594729 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:18.594709 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssdfp\" (UniqueName: \"kubernetes.io/projected/bb40b1b2-9e96-46d1-9e8e-f0586f3938b7-kube-api-access-ssdfp\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mgrpx\" (UID: \"bb40b1b2-9e96-46d1-9e8e-f0586f3938b7\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mgrpx" Jul 08 19:11:18.666385 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:18.666361 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mgrpx" Jul 08 19:11:18.752443 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:18.752412 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wc2k8"] Jul 08 19:11:18.755162 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:18.755140 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wc2k8" Jul 08 19:11:18.767909 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:18.767835 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wc2k8"] Jul 08 19:11:18.803422 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:18.801492 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mgrpx"] Jul 08 19:11:18.803878 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:11:18.803853 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb40b1b2_9e96_46d1_9e8e_f0586f3938b7.slice/crio-cefd6a61d3cccc1d9d8f022ed3fe243662d2c222f3925744a396a0db14eb76e0 WatchSource:0}: Error finding container cefd6a61d3cccc1d9d8f022ed3fe243662d2c222f3925744a396a0db14eb76e0: Status 404 returned error can't find the container with id cefd6a61d3cccc1d9d8f022ed3fe243662d2c222f3925744a396a0db14eb76e0 Jul 08 19:11:18.888026 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:18.887988 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7crtw\" (UniqueName: \"kubernetes.io/projected/bb55904d-f1a5-4a12-b403-d54a91df7de2-kube-api-access-7crtw\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wc2k8\" (UID: \"bb55904d-f1a5-4a12-b403-d54a91df7de2\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wc2k8" Jul 08 19:11:18.888026 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:18.888033 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb55904d-f1a5-4a12-b403-d54a91df7de2-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wc2k8\" (UID: \"bb55904d-f1a5-4a12-b403-d54a91df7de2\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wc2k8" Jul 08 19:11:18.888223 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:18.888100 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb55904d-f1a5-4a12-b403-d54a91df7de2-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wc2k8\" (UID: \"bb55904d-f1a5-4a12-b403-d54a91df7de2\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wc2k8" Jul 08 19:11:18.989371 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:18.989292 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7crtw\" (UniqueName: \"kubernetes.io/projected/bb55904d-f1a5-4a12-b403-d54a91df7de2-kube-api-access-7crtw\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wc2k8\" (UID: \"bb55904d-f1a5-4a12-b403-d54a91df7de2\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wc2k8" Jul 08 19:11:18.989371 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:18.989333 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb55904d-f1a5-4a12-b403-d54a91df7de2-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wc2k8\" (UID: \"bb55904d-f1a5-4a12-b403-d54a91df7de2\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wc2k8" Jul 08 19:11:18.989371 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:18.989368 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb55904d-f1a5-4a12-b403-d54a91df7de2-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wc2k8\" (UID: \"bb55904d-f1a5-4a12-b403-d54a91df7de2\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wc2k8" Jul 08 19:11:18.989772 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:18.989755 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb55904d-f1a5-4a12-b403-d54a91df7de2-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wc2k8\" (UID: \"bb55904d-f1a5-4a12-b403-d54a91df7de2\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wc2k8" Jul 08 19:11:18.989819 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:18.989766 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb55904d-f1a5-4a12-b403-d54a91df7de2-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wc2k8\" (UID: \"bb55904d-f1a5-4a12-b403-d54a91df7de2\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wc2k8" Jul 08 19:11:18.998516 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:18.998494 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7crtw\" (UniqueName: \"kubernetes.io/projected/bb55904d-f1a5-4a12-b403-d54a91df7de2-kube-api-access-7crtw\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wc2k8\" (UID: \"bb55904d-f1a5-4a12-b403-d54a91df7de2\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wc2k8" Jul 08 19:11:19.068418 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:19.068382 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wc2k8" Jul 08 19:11:19.155670 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:19.155630 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739vgr6"] Jul 08 19:11:19.159046 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:19.159024 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739vgr6" Jul 08 19:11:19.173242 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:19.173213 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739vgr6"] Jul 08 19:11:19.204131 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:19.204108 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wc2k8"] Jul 08 19:11:19.205547 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:11:19.205524 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb55904d_f1a5_4a12_b403_d54a91df7de2.slice/crio-3cfc5aa8b02467f3f01dd6be9a036b38b077189b16845dfddcf9105ea381be2b WatchSource:0}: Error finding container 3cfc5aa8b02467f3f01dd6be9a036b38b077189b16845dfddcf9105ea381be2b: Status 404 returned error can't find the container with id 3cfc5aa8b02467f3f01dd6be9a036b38b077189b16845dfddcf9105ea381be2b Jul 08 19:11:19.282552 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:19.282513 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wc2k8" event={"ID":"bb55904d-f1a5-4a12-b403-d54a91df7de2","Type":"ContainerStarted","Data":"6554de3b40c0fa82fb9e1699f312d4704a2b1866ea28c7fff22a5cb185115402"} Jul 08 19:11:19.282734 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:19.282566 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wc2k8" event={"ID":"bb55904d-f1a5-4a12-b403-d54a91df7de2","Type":"ContainerStarted","Data":"3cfc5aa8b02467f3f01dd6be9a036b38b077189b16845dfddcf9105ea381be2b"} Jul 08 19:11:19.284054 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:19.284020 2558 generic.go:356] "Generic (PLEG): container finished" podID="bb40b1b2-9e96-46d1-9e8e-f0586f3938b7" containerID="25b201002dd2615638769b1411e924f10b36ed2406d25d0083be5f855803581a" exitCode=0 Jul 08 19:11:19.284169 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:19.284074 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mgrpx" event={"ID":"bb40b1b2-9e96-46d1-9e8e-f0586f3938b7","Type":"ContainerDied","Data":"25b201002dd2615638769b1411e924f10b36ed2406d25d0083be5f855803581a"} Jul 08 19:11:19.284169 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:19.284117 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mgrpx" event={"ID":"bb40b1b2-9e96-46d1-9e8e-f0586f3938b7","Type":"ContainerStarted","Data":"cefd6a61d3cccc1d9d8f022ed3fe243662d2c222f3925744a396a0db14eb76e0"} Jul 08 19:11:19.285880 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:19.285854 2558 generic.go:356] "Generic (PLEG): container finished" podID="c24b2a5d-6bf4-4d86-a669-bcb498b921be" containerID="4e25b2d56e852f86106f8d89790527aa35cf5cc8bfaab28e77028c11c3bf9fdd" exitCode=0 Jul 08 19:11:19.285999 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:19.285890 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lqtlg" event={"ID":"c24b2a5d-6bf4-4d86-a669-bcb498b921be","Type":"ContainerDied","Data":"4e25b2d56e852f86106f8d89790527aa35cf5cc8bfaab28e77028c11c3bf9fdd"} Jul 08 19:11:19.291888 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:19.291865 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e758504b-8797-4466-9e69-b639a4694e6d-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739vgr6\" (UID: \"e758504b-8797-4466-9e69-b639a4694e6d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739vgr6" Jul 08 19:11:19.291995 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:19.291913 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45l94\" (UniqueName: \"kubernetes.io/projected/e758504b-8797-4466-9e69-b639a4694e6d-kube-api-access-45l94\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739vgr6\" (UID: \"e758504b-8797-4466-9e69-b639a4694e6d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739vgr6" Jul 08 19:11:19.292044 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:19.292005 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e758504b-8797-4466-9e69-b639a4694e6d-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739vgr6\" (UID: \"e758504b-8797-4466-9e69-b639a4694e6d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739vgr6" Jul 08 19:11:19.392498 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:19.392461 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45l94\" (UniqueName: \"kubernetes.io/projected/e758504b-8797-4466-9e69-b639a4694e6d-kube-api-access-45l94\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739vgr6\" (UID: \"e758504b-8797-4466-9e69-b639a4694e6d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739vgr6" Jul 08 19:11:19.392723 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:19.392545 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e758504b-8797-4466-9e69-b639a4694e6d-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739vgr6\" (UID: \"e758504b-8797-4466-9e69-b639a4694e6d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739vgr6" Jul 08 19:11:19.392723 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:19.392644 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e758504b-8797-4466-9e69-b639a4694e6d-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739vgr6\" (UID: \"e758504b-8797-4466-9e69-b639a4694e6d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739vgr6" Jul 08 19:11:19.393047 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:19.393025 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e758504b-8797-4466-9e69-b639a4694e6d-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739vgr6\" (UID: \"e758504b-8797-4466-9e69-b639a4694e6d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739vgr6" Jul 08 19:11:19.393138 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:19.393118 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e758504b-8797-4466-9e69-b639a4694e6d-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739vgr6\" (UID: \"e758504b-8797-4466-9e69-b639a4694e6d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739vgr6" Jul 08 19:11:19.401781 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:19.401755 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45l94\" (UniqueName: \"kubernetes.io/projected/e758504b-8797-4466-9e69-b639a4694e6d-kube-api-access-45l94\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739vgr6\" (UID: \"e758504b-8797-4466-9e69-b639a4694e6d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739vgr6" Jul 08 19:11:19.469383 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:19.469352 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739vgr6" Jul 08 19:11:19.602238 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:19.602171 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739vgr6"] Jul 08 19:11:19.604433 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:11:19.604402 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode758504b_8797_4466_9e69_b639a4694e6d.slice/crio-d74a5c1055167700570d72eb3d9118dbf37cf586f3c5e7de7db228d5467f1fcd WatchSource:0}: Error finding container d74a5c1055167700570d72eb3d9118dbf37cf586f3c5e7de7db228d5467f1fcd: Status 404 returned error can't find the container with id d74a5c1055167700570d72eb3d9118dbf37cf586f3c5e7de7db228d5467f1fcd Jul 08 19:11:20.291961 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:20.291930 2558 generic.go:356] "Generic (PLEG): container finished" podID="c24b2a5d-6bf4-4d86-a669-bcb498b921be" containerID="e8596144c500b695707497ec28c654a0872a05bfd0e92fab099420f94a6df6e5" exitCode=0 Jul 08 19:11:20.292373 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:20.292020 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lqtlg" event={"ID":"c24b2a5d-6bf4-4d86-a669-bcb498b921be","Type":"ContainerDied","Data":"e8596144c500b695707497ec28c654a0872a05bfd0e92fab099420f94a6df6e5"} Jul 08 19:11:20.293616 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:20.293569 2558 generic.go:356] "Generic (PLEG): container finished" podID="bb55904d-f1a5-4a12-b403-d54a91df7de2" containerID="6554de3b40c0fa82fb9e1699f312d4704a2b1866ea28c7fff22a5cb185115402" exitCode=0 Jul 08 19:11:20.293616 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:20.293604 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wc2k8" event={"ID":"bb55904d-f1a5-4a12-b403-d54a91df7de2","Type":"ContainerDied","Data":"6554de3b40c0fa82fb9e1699f312d4704a2b1866ea28c7fff22a5cb185115402"} Jul 08 19:11:20.295596 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:20.295561 2558 generic.go:356] "Generic (PLEG): container finished" podID="e758504b-8797-4466-9e69-b639a4694e6d" containerID="ee9a277cabc84334823da60955ef694aeae571e5d544e3bea7507ac55e2fbc50" exitCode=0 Jul 08 19:11:20.295708 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:20.295613 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739vgr6" event={"ID":"e758504b-8797-4466-9e69-b639a4694e6d","Type":"ContainerDied","Data":"ee9a277cabc84334823da60955ef694aeae571e5d544e3bea7507ac55e2fbc50"} Jul 08 19:11:20.295836 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:20.295707 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739vgr6" event={"ID":"e758504b-8797-4466-9e69-b639a4694e6d","Type":"ContainerStarted","Data":"d74a5c1055167700570d72eb3d9118dbf37cf586f3c5e7de7db228d5467f1fcd"} Jul 08 19:11:20.297645 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:20.297621 2558 generic.go:356] "Generic (PLEG): container finished" podID="bb40b1b2-9e96-46d1-9e8e-f0586f3938b7" containerID="1682fb20b7f1206da59e8d400b2dcd9423c9e3bffffe355d4d6a620f6a4ba93b" exitCode=0 Jul 08 19:11:20.297750 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:20.297681 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mgrpx" event={"ID":"bb40b1b2-9e96-46d1-9e8e-f0586f3938b7","Type":"ContainerDied","Data":"1682fb20b7f1206da59e8d400b2dcd9423c9e3bffffe355d4d6a620f6a4ba93b"} Jul 08 19:11:21.304276 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:21.304239 2558 generic.go:356] "Generic (PLEG): container finished" podID="bb40b1b2-9e96-46d1-9e8e-f0586f3938b7" containerID="62a28846d4b9dd67a568357ea483959c5b88f14d9baf92413883b5b70ca4be54" exitCode=0 Jul 08 19:11:21.304645 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:21.304312 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mgrpx" event={"ID":"bb40b1b2-9e96-46d1-9e8e-f0586f3938b7","Type":"ContainerDied","Data":"62a28846d4b9dd67a568357ea483959c5b88f14d9baf92413883b5b70ca4be54"} Jul 08 19:11:21.306346 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:21.306320 2558 generic.go:356] "Generic (PLEG): container finished" podID="c24b2a5d-6bf4-4d86-a669-bcb498b921be" containerID="4930b53964819e1ed59597b12fcf7bab3aa76405266d336e2bc7aba48ab89736" exitCode=0 Jul 08 19:11:21.306475 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:21.306401 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lqtlg" event={"ID":"c24b2a5d-6bf4-4d86-a669-bcb498b921be","Type":"ContainerDied","Data":"4930b53964819e1ed59597b12fcf7bab3aa76405266d336e2bc7aba48ab89736"} Jul 08 19:11:21.308270 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:21.308248 2558 generic.go:356] "Generic (PLEG): container finished" podID="bb55904d-f1a5-4a12-b403-d54a91df7de2" containerID="68bc7c2247c19d6471daa4a579892da6e8b0a631f12ce7ac4742643483c746d0" exitCode=0 Jul 08 19:11:21.308386 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:21.308332 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wc2k8" event={"ID":"bb55904d-f1a5-4a12-b403-d54a91df7de2","Type":"ContainerDied","Data":"68bc7c2247c19d6471daa4a579892da6e8b0a631f12ce7ac4742643483c746d0"} Jul 08 19:11:21.310233 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:21.310209 2558 generic.go:356] "Generic (PLEG): container finished" podID="e758504b-8797-4466-9e69-b639a4694e6d" containerID="11fe50380827bbaafdba2f982515147d8cceea1377e6643e2f40454700ff628f" exitCode=0 Jul 08 19:11:21.310389 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:21.310262 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739vgr6" event={"ID":"e758504b-8797-4466-9e69-b639a4694e6d","Type":"ContainerDied","Data":"11fe50380827bbaafdba2f982515147d8cceea1377e6643e2f40454700ff628f"} Jul 08 19:11:22.316708 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:22.316620 2558 generic.go:356] "Generic (PLEG): container finished" podID="bb55904d-f1a5-4a12-b403-d54a91df7de2" containerID="4a2164c4ac526006f4b2714d9ebd483cec6c7c83c68d2a7f67c5beeb0ccf3965" exitCode=0 Jul 08 19:11:22.317102 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:22.316704 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wc2k8" event={"ID":"bb55904d-f1a5-4a12-b403-d54a91df7de2","Type":"ContainerDied","Data":"4a2164c4ac526006f4b2714d9ebd483cec6c7c83c68d2a7f67c5beeb0ccf3965"} Jul 08 19:11:22.318837 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:22.318809 2558 generic.go:356] "Generic (PLEG): container finished" podID="e758504b-8797-4466-9e69-b639a4694e6d" containerID="98656f93308f42a22f5541243cae59e19a944f0774c3b9ad52bbef9a351b27da" exitCode=0 Jul 08 19:11:22.318946 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:22.318926 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739vgr6" event={"ID":"e758504b-8797-4466-9e69-b639a4694e6d","Type":"ContainerDied","Data":"98656f93308f42a22f5541243cae59e19a944f0774c3b9ad52bbef9a351b27da"} Jul 08 19:11:22.467731 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:22.467702 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mgrpx" Jul 08 19:11:22.470833 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:22.470812 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lqtlg" Jul 08 19:11:22.620163 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:22.620131 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb40b1b2-9e96-46d1-9e8e-f0586f3938b7-util\") pod \"bb40b1b2-9e96-46d1-9e8e-f0586f3938b7\" (UID: \"bb40b1b2-9e96-46d1-9e8e-f0586f3938b7\") " Jul 08 19:11:22.620314 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:22.620181 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c24b2a5d-6bf4-4d86-a669-bcb498b921be-bundle\") pod \"c24b2a5d-6bf4-4d86-a669-bcb498b921be\" (UID: \"c24b2a5d-6bf4-4d86-a669-bcb498b921be\") " Jul 08 19:11:22.620314 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:22.620250 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ssdfp\" (UniqueName: \"kubernetes.io/projected/bb40b1b2-9e96-46d1-9e8e-f0586f3938b7-kube-api-access-ssdfp\") pod \"bb40b1b2-9e96-46d1-9e8e-f0586f3938b7\" (UID: \"bb40b1b2-9e96-46d1-9e8e-f0586f3938b7\") " Jul 08 19:11:22.620314 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:22.620293 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c24b2a5d-6bf4-4d86-a669-bcb498b921be-util\") pod \"c24b2a5d-6bf4-4d86-a669-bcb498b921be\" (UID: \"c24b2a5d-6bf4-4d86-a669-bcb498b921be\") " Jul 08 19:11:22.620442 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:22.620316 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb40b1b2-9e96-46d1-9e8e-f0586f3938b7-bundle\") pod \"bb40b1b2-9e96-46d1-9e8e-f0586f3938b7\" (UID: \"bb40b1b2-9e96-46d1-9e8e-f0586f3938b7\") " Jul 08 19:11:22.620442 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:22.620347 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s52tj\" (UniqueName: \"kubernetes.io/projected/c24b2a5d-6bf4-4d86-a669-bcb498b921be-kube-api-access-s52tj\") pod \"c24b2a5d-6bf4-4d86-a669-bcb498b921be\" (UID: \"c24b2a5d-6bf4-4d86-a669-bcb498b921be\") " Jul 08 19:11:22.620951 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:22.620916 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb40b1b2-9e96-46d1-9e8e-f0586f3938b7-bundle" (OuterVolumeSpecName: "bundle") pod "bb40b1b2-9e96-46d1-9e8e-f0586f3938b7" (UID: "bb40b1b2-9e96-46d1-9e8e-f0586f3938b7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:11:22.620951 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:22.620925 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c24b2a5d-6bf4-4d86-a669-bcb498b921be-bundle" (OuterVolumeSpecName: "bundle") pod "c24b2a5d-6bf4-4d86-a669-bcb498b921be" (UID: "c24b2a5d-6bf4-4d86-a669-bcb498b921be"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:11:22.622554 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:22.622529 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb40b1b2-9e96-46d1-9e8e-f0586f3938b7-kube-api-access-ssdfp" (OuterVolumeSpecName: "kube-api-access-ssdfp") pod "bb40b1b2-9e96-46d1-9e8e-f0586f3938b7" (UID: "bb40b1b2-9e96-46d1-9e8e-f0586f3938b7"). InnerVolumeSpecName "kube-api-access-ssdfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:11:22.622911 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:22.622887 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c24b2a5d-6bf4-4d86-a669-bcb498b921be-kube-api-access-s52tj" (OuterVolumeSpecName: "kube-api-access-s52tj") pod "c24b2a5d-6bf4-4d86-a669-bcb498b921be" (UID: "c24b2a5d-6bf4-4d86-a669-bcb498b921be"). InnerVolumeSpecName "kube-api-access-s52tj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:11:22.626323 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:22.626289 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c24b2a5d-6bf4-4d86-a669-bcb498b921be-util" (OuterVolumeSpecName: "util") pod "c24b2a5d-6bf4-4d86-a669-bcb498b921be" (UID: "c24b2a5d-6bf4-4d86-a669-bcb498b921be"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:11:22.626738 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:22.626718 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb40b1b2-9e96-46d1-9e8e-f0586f3938b7-util" (OuterVolumeSpecName: "util") pod "bb40b1b2-9e96-46d1-9e8e-f0586f3938b7" (UID: "bb40b1b2-9e96-46d1-9e8e-f0586f3938b7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:11:22.721011 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:22.720961 2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c24b2a5d-6bf4-4d86-a669-bcb498b921be-util\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:11:22.721011 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:22.721006 2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb40b1b2-9e96-46d1-9e8e-f0586f3938b7-bundle\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:11:22.721011 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:22.721019 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s52tj\" (UniqueName: \"kubernetes.io/projected/c24b2a5d-6bf4-4d86-a669-bcb498b921be-kube-api-access-s52tj\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:11:22.721246 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:22.721032 2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb40b1b2-9e96-46d1-9e8e-f0586f3938b7-util\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:11:22.721246 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:22.721045 2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c24b2a5d-6bf4-4d86-a669-bcb498b921be-bundle\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:11:22.721246 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:22.721055 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssdfp\" (UniqueName: \"kubernetes.io/projected/bb40b1b2-9e96-46d1-9e8e-f0586f3938b7-kube-api-access-ssdfp\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:11:23.325743 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:23.325715 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mgrpx" Jul 08 19:11:23.325743 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:23.325731 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mgrpx" event={"ID":"bb40b1b2-9e96-46d1-9e8e-f0586f3938b7","Type":"ContainerDied","Data":"cefd6a61d3cccc1d9d8f022ed3fe243662d2c222f3925744a396a0db14eb76e0"} Jul 08 19:11:23.326140 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:23.325785 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cefd6a61d3cccc1d9d8f022ed3fe243662d2c222f3925744a396a0db14eb76e0" Jul 08 19:11:23.327919 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:23.327892 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lqtlg" event={"ID":"c24b2a5d-6bf4-4d86-a669-bcb498b921be","Type":"ContainerDied","Data":"490804b006159139a1cafc5b754b1650c794a349c91ecf65a0ee4c6b82c8f162"} Jul 08 19:11:23.328057 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:23.327929 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="490804b006159139a1cafc5b754b1650c794a349c91ecf65a0ee4c6b82c8f162" Jul 08 19:11:23.328057 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:23.327931 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lqtlg" Jul 08 19:11:23.482564 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:23.482539 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wc2k8" Jul 08 19:11:23.485203 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:23.485177 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739vgr6" Jul 08 19:11:23.628464 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:23.628437 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45l94\" (UniqueName: \"kubernetes.io/projected/e758504b-8797-4466-9e69-b639a4694e6d-kube-api-access-45l94\") pod \"e758504b-8797-4466-9e69-b639a4694e6d\" (UID: \"e758504b-8797-4466-9e69-b639a4694e6d\") " Jul 08 19:11:23.628636 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:23.628492 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7crtw\" (UniqueName: \"kubernetes.io/projected/bb55904d-f1a5-4a12-b403-d54a91df7de2-kube-api-access-7crtw\") pod \"bb55904d-f1a5-4a12-b403-d54a91df7de2\" (UID: \"bb55904d-f1a5-4a12-b403-d54a91df7de2\") " Jul 08 19:11:23.628636 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:23.628528 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb55904d-f1a5-4a12-b403-d54a91df7de2-util\") pod \"bb55904d-f1a5-4a12-b403-d54a91df7de2\" (UID: \"bb55904d-f1a5-4a12-b403-d54a91df7de2\") " Jul 08 19:11:23.628636 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:23.628556 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e758504b-8797-4466-9e69-b639a4694e6d-bundle\") pod \"e758504b-8797-4466-9e69-b639a4694e6d\" (UID: \"e758504b-8797-4466-9e69-b639a4694e6d\") " Jul 08 19:11:23.628636 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:23.628603 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb55904d-f1a5-4a12-b403-d54a91df7de2-bundle\") pod \"bb55904d-f1a5-4a12-b403-d54a91df7de2\" (UID: \"bb55904d-f1a5-4a12-b403-d54a91df7de2\") " Jul 08 19:11:23.628874 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:23.628638 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e758504b-8797-4466-9e69-b639a4694e6d-util\") pod \"e758504b-8797-4466-9e69-b639a4694e6d\" (UID: \"e758504b-8797-4466-9e69-b639a4694e6d\") " Jul 08 19:11:23.629152 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:23.629130 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb55904d-f1a5-4a12-b403-d54a91df7de2-bundle" (OuterVolumeSpecName: "bundle") pod "bb55904d-f1a5-4a12-b403-d54a91df7de2" (UID: "bb55904d-f1a5-4a12-b403-d54a91df7de2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:11:23.629461 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:23.629434 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e758504b-8797-4466-9e69-b639a4694e6d-bundle" (OuterVolumeSpecName: "bundle") pod "e758504b-8797-4466-9e69-b639a4694e6d" (UID: "e758504b-8797-4466-9e69-b639a4694e6d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:11:23.630836 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:23.630816 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e758504b-8797-4466-9e69-b639a4694e6d-kube-api-access-45l94" (OuterVolumeSpecName: "kube-api-access-45l94") pod "e758504b-8797-4466-9e69-b639a4694e6d" (UID: "e758504b-8797-4466-9e69-b639a4694e6d"). InnerVolumeSpecName "kube-api-access-45l94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:11:23.631063 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:23.631039 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb55904d-f1a5-4a12-b403-d54a91df7de2-kube-api-access-7crtw" (OuterVolumeSpecName: "kube-api-access-7crtw") pod "bb55904d-f1a5-4a12-b403-d54a91df7de2" (UID: "bb55904d-f1a5-4a12-b403-d54a91df7de2"). InnerVolumeSpecName "kube-api-access-7crtw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:11:23.634003 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:23.633971 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e758504b-8797-4466-9e69-b639a4694e6d-util" (OuterVolumeSpecName: "util") pod "e758504b-8797-4466-9e69-b639a4694e6d" (UID: "e758504b-8797-4466-9e69-b639a4694e6d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:11:23.634176 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:23.634158 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb55904d-f1a5-4a12-b403-d54a91df7de2-util" (OuterVolumeSpecName: "util") pod "bb55904d-f1a5-4a12-b403-d54a91df7de2" (UID: "bb55904d-f1a5-4a12-b403-d54a91df7de2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:11:23.730216 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:23.730183 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45l94\" (UniqueName: \"kubernetes.io/projected/e758504b-8797-4466-9e69-b639a4694e6d-kube-api-access-45l94\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:11:23.730216 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:23.730211 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7crtw\" (UniqueName: \"kubernetes.io/projected/bb55904d-f1a5-4a12-b403-d54a91df7de2-kube-api-access-7crtw\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:11:23.730216 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:23.730220 2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb55904d-f1a5-4a12-b403-d54a91df7de2-util\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:11:23.730216 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:23.730229 2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e758504b-8797-4466-9e69-b639a4694e6d-bundle\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:11:23.730482 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:23.730237 2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb55904d-f1a5-4a12-b403-d54a91df7de2-bundle\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:11:23.730482 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:23.730244 2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e758504b-8797-4466-9e69-b639a4694e6d-util\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:11:24.335328 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:24.335289 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wc2k8" event={"ID":"bb55904d-f1a5-4a12-b403-d54a91df7de2","Type":"ContainerDied","Data":"3cfc5aa8b02467f3f01dd6be9a036b38b077189b16845dfddcf9105ea381be2b"} Jul 08 19:11:24.335791 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:24.335342 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cfc5aa8b02467f3f01dd6be9a036b38b077189b16845dfddcf9105ea381be2b" Jul 08 19:11:24.335791 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:24.335301 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wc2k8" Jul 08 19:11:24.337171 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:24.337145 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739vgr6" event={"ID":"e758504b-8797-4466-9e69-b639a4694e6d","Type":"ContainerDied","Data":"d74a5c1055167700570d72eb3d9118dbf37cf586f3c5e7de7db228d5467f1fcd"} Jul 08 19:11:24.337171 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:24.337169 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739vgr6" Jul 08 19:11:24.337327 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:24.337189 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d74a5c1055167700570d72eb3d9118dbf37cf586f3c5e7de7db228d5467f1fcd" Jul 08 19:11:30.676873 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:30.676841 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:11:31.274320 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:31.274278 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:11:32.358562 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:32.358523 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:11:33.743668 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:33.743626 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:11:33.849025 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:33.848992 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:11:35.923441 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:35.923395 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:11:38.296982 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:38.296941 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:11:39.264746 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:39.264713 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:11:40.569561 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:40.569522 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:11:42.165675 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:42.165638 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:11:42.758921 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:42.758879 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:11:44.998433 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:44.998391 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:11:45.096563 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:45.096534 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:11:45.301921 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:45.301826 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:11:45.409320 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:45.409278 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:11:45.525845 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:45.525808 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:11:45.731203 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:45.731172 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:11:45.862662 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:45.862631 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:11:49.511039 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:49.511002 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:11:49.612738 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:49.612699 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:11:50.444900 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:50.444865 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:11:50.542970 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:50.542936 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:11:52.713519 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:52.713478 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:11:52.813982 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:52.813946 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:11:55.275384 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:55.275349 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6dfcdb7b79-9trk6"] Jul 08 19:11:55.496375 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:55.496344 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:11:55.626545 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:55.626513 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:11:55.741776 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:55.741738 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:11:59.457913 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:59.457882 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:11:59.569516 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:11:59.569479 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:12:04.702430 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:04.702388 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:12:04.846938 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:04.846906 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:12:05.305922 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:05.305887 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:12:05.447015 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:05.446970 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:12:09.356724 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:09.356687 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:12:09.512883 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:09.512825 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:12:12.370768 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:12.370727 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:12:12.900079 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:12.900049 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:12:18.938014 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:18.937982 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:12:19.210429 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:19.210328 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:12:20.295615 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:20.295534 2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6dfcdb7b79-9trk6" podUID="4693f8b5-7900-4c88-98de-2d26bed5b8c5" containerName="console" containerID="cri-o://8ce269f922ce60c25f3f1d45c2ba1c8bd74632dbf3d754ca9d0009e5c87ce85e" gracePeriod=15 Jul 08 19:12:20.539681 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:20.539658 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dfcdb7b79-9trk6" Jul 08 19:12:20.608679 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:20.608561 2558 generic.go:356] "Generic (PLEG): container finished" podID="4693f8b5-7900-4c88-98de-2d26bed5b8c5" containerID="8ce269f922ce60c25f3f1d45c2ba1c8bd74632dbf3d754ca9d0009e5c87ce85e" exitCode=0 Jul 08 19:12:20.608679 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:20.608621 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dfcdb7b79-9trk6" Jul 08 19:12:20.608842 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:20.608621 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dfcdb7b79-9trk6" event={"ID":"4693f8b5-7900-4c88-98de-2d26bed5b8c5","Type":"ContainerDied","Data":"8ce269f922ce60c25f3f1d45c2ba1c8bd74632dbf3d754ca9d0009e5c87ce85e"} Jul 08 19:12:20.608842 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:20.608737 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dfcdb7b79-9trk6" event={"ID":"4693f8b5-7900-4c88-98de-2d26bed5b8c5","Type":"ContainerDied","Data":"43aa8e213189aca47ea03b1a562d85fff7e2e23d6d254d776558ec4d852f0461"} Jul 08 19:12:20.608842 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:20.608768 2558 scope.go:117] "RemoveContainer" containerID="8ce269f922ce60c25f3f1d45c2ba1c8bd74632dbf3d754ca9d0009e5c87ce85e" Jul 08 19:12:20.617301 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:20.617284 2558 scope.go:117] "RemoveContainer" containerID="8ce269f922ce60c25f3f1d45c2ba1c8bd74632dbf3d754ca9d0009e5c87ce85e" Jul 08 19:12:20.617606 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:12:20.617563 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ce269f922ce60c25f3f1d45c2ba1c8bd74632dbf3d754ca9d0009e5c87ce85e\": container with ID starting with 8ce269f922ce60c25f3f1d45c2ba1c8bd74632dbf3d754ca9d0009e5c87ce85e not found: ID does not exist" containerID="8ce269f922ce60c25f3f1d45c2ba1c8bd74632dbf3d754ca9d0009e5c87ce85e" Jul 08 19:12:20.617704 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:20.617614 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ce269f922ce60c25f3f1d45c2ba1c8bd74632dbf3d754ca9d0009e5c87ce85e"} err="failed to get container status \"8ce269f922ce60c25f3f1d45c2ba1c8bd74632dbf3d754ca9d0009e5c87ce85e\": rpc error: code = NotFound desc = could not find container \"8ce269f922ce60c25f3f1d45c2ba1c8bd74632dbf3d754ca9d0009e5c87ce85e\": container with ID starting with 8ce269f922ce60c25f3f1d45c2ba1c8bd74632dbf3d754ca9d0009e5c87ce85e not found: ID does not exist" Jul 08 19:12:20.696618 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:20.696569 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4693f8b5-7900-4c88-98de-2d26bed5b8c5-oauth-serving-cert\") pod \"4693f8b5-7900-4c88-98de-2d26bed5b8c5\" (UID: \"4693f8b5-7900-4c88-98de-2d26bed5b8c5\") " Jul 08 19:12:20.696799 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:20.696635 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4693f8b5-7900-4c88-98de-2d26bed5b8c5-console-oauth-config\") pod \"4693f8b5-7900-4c88-98de-2d26bed5b8c5\" (UID: \"4693f8b5-7900-4c88-98de-2d26bed5b8c5\") " Jul 08 19:12:20.696799 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:20.696673 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4693f8b5-7900-4c88-98de-2d26bed5b8c5-console-serving-cert\") pod \"4693f8b5-7900-4c88-98de-2d26bed5b8c5\" (UID: \"4693f8b5-7900-4c88-98de-2d26bed5b8c5\") " Jul 08 19:12:20.696920 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:20.696818 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4693f8b5-7900-4c88-98de-2d26bed5b8c5-console-config\") pod \"4693f8b5-7900-4c88-98de-2d26bed5b8c5\" (UID: \"4693f8b5-7900-4c88-98de-2d26bed5b8c5\") " Jul 08 19:12:20.696920 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:20.696850 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4693f8b5-7900-4c88-98de-2d26bed5b8c5-trusted-ca-bundle\") pod \"4693f8b5-7900-4c88-98de-2d26bed5b8c5\" (UID: \"4693f8b5-7900-4c88-98de-2d26bed5b8c5\") " Jul 08 19:12:20.696920 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:20.696878 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4693f8b5-7900-4c88-98de-2d26bed5b8c5-service-ca\") pod \"4693f8b5-7900-4c88-98de-2d26bed5b8c5\" (UID: \"4693f8b5-7900-4c88-98de-2d26bed5b8c5\") " Jul 08 19:12:20.696920 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:20.696910 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cm4h8\" (UniqueName: \"kubernetes.io/projected/4693f8b5-7900-4c88-98de-2d26bed5b8c5-kube-api-access-cm4h8\") pod \"4693f8b5-7900-4c88-98de-2d26bed5b8c5\" (UID: \"4693f8b5-7900-4c88-98de-2d26bed5b8c5\") " Jul 08 19:12:20.697124 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:20.697004 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4693f8b5-7900-4c88-98de-2d26bed5b8c5-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4693f8b5-7900-4c88-98de-2d26bed5b8c5" (UID: "4693f8b5-7900-4c88-98de-2d26bed5b8c5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:12:20.697208 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:20.697186 2558 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4693f8b5-7900-4c88-98de-2d26bed5b8c5-oauth-serving-cert\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:12:20.697258 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:20.697183 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4693f8b5-7900-4c88-98de-2d26bed5b8c5-console-config" (OuterVolumeSpecName: "console-config") pod "4693f8b5-7900-4c88-98de-2d26bed5b8c5" (UID: "4693f8b5-7900-4c88-98de-2d26bed5b8c5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:12:20.697258 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:20.697238 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4693f8b5-7900-4c88-98de-2d26bed5b8c5-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4693f8b5-7900-4c88-98de-2d26bed5b8c5" (UID: "4693f8b5-7900-4c88-98de-2d26bed5b8c5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:12:20.697346 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:20.697302 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4693f8b5-7900-4c88-98de-2d26bed5b8c5-service-ca" (OuterVolumeSpecName: "service-ca") pod "4693f8b5-7900-4c88-98de-2d26bed5b8c5" (UID: "4693f8b5-7900-4c88-98de-2d26bed5b8c5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:12:20.698885 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:20.698849 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4693f8b5-7900-4c88-98de-2d26bed5b8c5-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4693f8b5-7900-4c88-98de-2d26bed5b8c5" (UID: "4693f8b5-7900-4c88-98de-2d26bed5b8c5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:12:20.698885 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:20.698869 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4693f8b5-7900-4c88-98de-2d26bed5b8c5-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4693f8b5-7900-4c88-98de-2d26bed5b8c5" (UID: "4693f8b5-7900-4c88-98de-2d26bed5b8c5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:12:20.699060 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:20.698949 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4693f8b5-7900-4c88-98de-2d26bed5b8c5-kube-api-access-cm4h8" (OuterVolumeSpecName: "kube-api-access-cm4h8") pod "4693f8b5-7900-4c88-98de-2d26bed5b8c5" (UID: "4693f8b5-7900-4c88-98de-2d26bed5b8c5"). InnerVolumeSpecName "kube-api-access-cm4h8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:12:20.798234 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:20.798207 2558 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4693f8b5-7900-4c88-98de-2d26bed5b8c5-console-oauth-config\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:12:20.798234 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:20.798229 2558 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4693f8b5-7900-4c88-98de-2d26bed5b8c5-console-serving-cert\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:12:20.798234 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:20.798240 2558 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4693f8b5-7900-4c88-98de-2d26bed5b8c5-console-config\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:12:20.798425 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:20.798249 2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4693f8b5-7900-4c88-98de-2d26bed5b8c5-trusted-ca-bundle\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:12:20.798425 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:20.798258 2558 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4693f8b5-7900-4c88-98de-2d26bed5b8c5-service-ca\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:12:20.798425 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:20.798266 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm4h8\" (UniqueName: \"kubernetes.io/projected/4693f8b5-7900-4c88-98de-2d26bed5b8c5-kube-api-access-cm4h8\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:12:20.953398 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:20.953369 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6dfcdb7b79-9trk6"] Jul 08 19:12:20.973613 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:20.973586 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6dfcdb7b79-9trk6"] Jul 08 19:12:21.281302 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:21.281223 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4693f8b5-7900-4c88-98de-2d26bed5b8c5" path="/var/lib/kubelet/pods/4693f8b5-7900-4c88-98de-2d26bed5b8c5/volumes" Jul 08 19:12:34.864003 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:34.863920 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:12:35.319416 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:35.319383 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:12:37.348780 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.348746 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:12:37.358222 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.358196 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5d696"] Jul 08 19:12:37.358508 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.358497 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e758504b-8797-4466-9e69-b639a4694e6d" containerName="pull" Jul 08 19:12:37.358551 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.358510 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e758504b-8797-4466-9e69-b639a4694e6d" containerName="pull" Jul 08 19:12:37.358551 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.358521 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb40b1b2-9e96-46d1-9e8e-f0586f3938b7" containerName="pull" Jul 08 19:12:37.358551 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.358527 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb40b1b2-9e96-46d1-9e8e-f0586f3938b7" containerName="pull" Jul 08 19:12:37.358551 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.358533 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb55904d-f1a5-4a12-b403-d54a91df7de2" containerName="extract" Jul 08 19:12:37.358551 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.358538 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb55904d-f1a5-4a12-b403-d54a91df7de2" containerName="extract" Jul 08 19:12:37.358551 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.358550 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e758504b-8797-4466-9e69-b639a4694e6d" containerName="util" Jul 08 19:12:37.358752 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.358555 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e758504b-8797-4466-9e69-b639a4694e6d" containerName="util" Jul 08 19:12:37.358752 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.358563 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb55904d-f1a5-4a12-b403-d54a91df7de2" containerName="pull" Jul 08 19:12:37.358752 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.358568 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb55904d-f1a5-4a12-b403-d54a91df7de2" containerName="pull" Jul 08 19:12:37.358752 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.358590 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb55904d-f1a5-4a12-b403-d54a91df7de2" containerName="util" Jul 08 19:12:37.358752 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.358598 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb55904d-f1a5-4a12-b403-d54a91df7de2" containerName="util" Jul 08 19:12:37.358752 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.358608 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e758504b-8797-4466-9e69-b639a4694e6d" containerName="extract" Jul 08 19:12:37.358752 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.358616 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e758504b-8797-4466-9e69-b639a4694e6d" containerName="extract" Jul 08 19:12:37.358752 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.358627 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4693f8b5-7900-4c88-98de-2d26bed5b8c5" containerName="console" Jul 08 19:12:37.358752 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.358631 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4693f8b5-7900-4c88-98de-2d26bed5b8c5" containerName="console" Jul 08 19:12:37.358752 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.358637 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c24b2a5d-6bf4-4d86-a669-bcb498b921be" containerName="pull" Jul 08 19:12:37.358752 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.358642 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c24b2a5d-6bf4-4d86-a669-bcb498b921be" containerName="pull" Jul 08 19:12:37.358752 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.358647 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb40b1b2-9e96-46d1-9e8e-f0586f3938b7" containerName="extract" Jul 08 19:12:37.358752 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.358652 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb40b1b2-9e96-46d1-9e8e-f0586f3938b7" containerName="extract" Jul 08 19:12:37.358752 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.358660 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb40b1b2-9e96-46d1-9e8e-f0586f3938b7" containerName="util" Jul 08 19:12:37.358752 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.358667 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb40b1b2-9e96-46d1-9e8e-f0586f3938b7" containerName="util" Jul 08 19:12:37.358752 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.358674 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c24b2a5d-6bf4-4d86-a669-bcb498b921be" containerName="extract" Jul 08 19:12:37.358752 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.358681 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c24b2a5d-6bf4-4d86-a669-bcb498b921be" containerName="extract" Jul 08 19:12:37.358752 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.358691 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c24b2a5d-6bf4-4d86-a669-bcb498b921be" containerName="util" Jul 08 19:12:37.358752 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.358696 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c24b2a5d-6bf4-4d86-a669-bcb498b921be" containerName="util" Jul 08 19:12:37.358752 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.358750 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e758504b-8797-4466-9e69-b639a4694e6d" containerName="extract" Jul 08 19:12:37.358752 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.358757 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c24b2a5d-6bf4-4d86-a669-bcb498b921be" containerName="extract" Jul 08 19:12:37.359380 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.358782 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb40b1b2-9e96-46d1-9e8e-f0586f3938b7" containerName="extract" Jul 08 19:12:37.359380 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.358789 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb55904d-f1a5-4a12-b403-d54a91df7de2" containerName="extract" Jul 08 19:12:37.359380 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.358794 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4693f8b5-7900-4c88-98de-2d26bed5b8c5" containerName="console" Jul 08 19:12:37.362826 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.362792 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-5d696" Jul 08 19:12:37.365354 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.365329 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 08 19:12:37.365485 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.365395 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 08 19:12:37.365485 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.365400 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 08 19:12:37.366138 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.366122 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2vkt6\"" Jul 08 19:12:37.377203 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.377175 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5d696"] Jul 08 19:12:37.446977 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.446904 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5d696"] Jul 08 19:12:37.451298 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.451272 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:12:37.517495 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.517465 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:12:37.539270 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.539245 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts899\" (UniqueName: \"kubernetes.io/projected/7093efec-8b60-4b8f-821d-42e3c2b4fbe6-kube-api-access-ts899\") pod \"limitador-limitador-64c94fd64f-5d696\" (UID: \"7093efec-8b60-4b8f-821d-42e3c2b4fbe6\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-5d696" Jul 08 19:12:37.539403 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.539314 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7093efec-8b60-4b8f-821d-42e3c2b4fbe6-config-file\") pod \"limitador-limitador-64c94fd64f-5d696\" (UID: \"7093efec-8b60-4b8f-821d-42e3c2b4fbe6\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-5d696" Jul 08 19:12:37.639881 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.639841 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts899\" (UniqueName: \"kubernetes.io/projected/7093efec-8b60-4b8f-821d-42e3c2b4fbe6-kube-api-access-ts899\") pod \"limitador-limitador-64c94fd64f-5d696\" (UID: \"7093efec-8b60-4b8f-821d-42e3c2b4fbe6\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-5d696" Jul 08 19:12:37.640071 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.639902 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7093efec-8b60-4b8f-821d-42e3c2b4fbe6-config-file\") pod \"limitador-limitador-64c94fd64f-5d696\" (UID: \"7093efec-8b60-4b8f-821d-42e3c2b4fbe6\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-5d696" Jul 08 19:12:37.640527 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.640503 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7093efec-8b60-4b8f-821d-42e3c2b4fbe6-config-file\") pod \"limitador-limitador-64c94fd64f-5d696\" (UID: \"7093efec-8b60-4b8f-821d-42e3c2b4fbe6\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-5d696" Jul 08 19:12:37.648639 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.648612 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts899\" (UniqueName: \"kubernetes.io/projected/7093efec-8b60-4b8f-821d-42e3c2b4fbe6-kube-api-access-ts899\") pod \"limitador-limitador-64c94fd64f-5d696\" (UID: \"7093efec-8b60-4b8f-821d-42e3c2b4fbe6\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-5d696" Jul 08 19:12:37.674871 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.674846 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-5d696" Jul 08 19:12:37.815111 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:37.815085 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5d696"] Jul 08 19:12:37.817063 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:12:37.817037 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7093efec_8b60_4b8f_821d_42e3c2b4fbe6.slice/crio-63e9c7d8f61d811a7bbcf71a482f5ade0c1cbf2574e78f1bb9887202e04890b5 WatchSource:0}: Error finding container 63e9c7d8f61d811a7bbcf71a482f5ade0c1cbf2574e78f1bb9887202e04890b5: Status 404 returned error can't find the container with id 63e9c7d8f61d811a7bbcf71a482f5ade0c1cbf2574e78f1bb9887202e04890b5 Jul 08 19:12:38.205827 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:38.205791 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:12:38.307737 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:38.307701 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:12:38.701097 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:38.701056 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-5d696" event={"ID":"7093efec-8b60-4b8f-821d-42e3c2b4fbe6","Type":"ContainerStarted","Data":"63e9c7d8f61d811a7bbcf71a482f5ade0c1cbf2574e78f1bb9887202e04890b5"} Jul 08 19:12:40.713963 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:40.713867 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-5d696" event={"ID":"7093efec-8b60-4b8f-821d-42e3c2b4fbe6","Type":"ContainerStarted","Data":"10d4af923735d013cd80d0bc30439e7f37363f3d711d915c38b0dfb7fc883ee1"} Jul 08 19:12:40.713963 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:40.713948 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-5d696" Jul 08 19:12:40.737051 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:40.736988 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-5d696" podStartSLOduration=1.18084553 podStartE2EDuration="3.736952807s" totalImagesPullingTime="2.556107277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:12:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:12:40.73639422 +0000 UTC m=+543.995347146" watchObservedRunningTime="2026-07-08 19:12:40.736952807 +0000 UTC m=+543.995905739" Jul 08 19:12:51.719321 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:51.719287 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-5d696" Jul 08 19:12:52.603301 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:52.603269 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5d696"] Jul 08 19:12:52.603538 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:52.603465 2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-5d696" podUID="7093efec-8b60-4b8f-821d-42e3c2b4fbe6" containerName="limitador" containerID="cri-o://10d4af923735d013cd80d0bc30439e7f37363f3d711d915c38b0dfb7fc883ee1" gracePeriod=30 Jul 08 19:12:53.147562 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:53.147537 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-5d696" Jul 08 19:12:53.162254 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:53.162232 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7093efec-8b60-4b8f-821d-42e3c2b4fbe6-config-file\") pod \"7093efec-8b60-4b8f-821d-42e3c2b4fbe6\" (UID: \"7093efec-8b60-4b8f-821d-42e3c2b4fbe6\") " Jul 08 19:12:53.162385 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:53.162267 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ts899\" (UniqueName: \"kubernetes.io/projected/7093efec-8b60-4b8f-821d-42e3c2b4fbe6-kube-api-access-ts899\") pod \"7093efec-8b60-4b8f-821d-42e3c2b4fbe6\" (UID: \"7093efec-8b60-4b8f-821d-42e3c2b4fbe6\") " Jul 08 19:12:53.162613 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:53.162569 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7093efec-8b60-4b8f-821d-42e3c2b4fbe6-config-file" (OuterVolumeSpecName: "config-file") pod "7093efec-8b60-4b8f-821d-42e3c2b4fbe6" (UID: "7093efec-8b60-4b8f-821d-42e3c2b4fbe6"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:12:53.164287 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:53.164238 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7093efec-8b60-4b8f-821d-42e3c2b4fbe6-kube-api-access-ts899" (OuterVolumeSpecName: "kube-api-access-ts899") pod "7093efec-8b60-4b8f-821d-42e3c2b4fbe6" (UID: "7093efec-8b60-4b8f-821d-42e3c2b4fbe6"). InnerVolumeSpecName "kube-api-access-ts899". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:12:53.263732 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:53.263705 2558 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7093efec-8b60-4b8f-821d-42e3c2b4fbe6-config-file\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:12:53.263732 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:53.263730 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts899\" (UniqueName: \"kubernetes.io/projected/7093efec-8b60-4b8f-821d-42e3c2b4fbe6-kube-api-access-ts899\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:12:53.774879 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:53.774849 2558 generic.go:356] "Generic (PLEG): container finished" podID="7093efec-8b60-4b8f-821d-42e3c2b4fbe6" containerID="10d4af923735d013cd80d0bc30439e7f37363f3d711d915c38b0dfb7fc883ee1" exitCode=0 Jul 08 19:12:53.775036 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:53.774897 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-5d696" Jul 08 19:12:53.775036 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:53.774929 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-5d696" event={"ID":"7093efec-8b60-4b8f-821d-42e3c2b4fbe6","Type":"ContainerDied","Data":"10d4af923735d013cd80d0bc30439e7f37363f3d711d915c38b0dfb7fc883ee1"} Jul 08 19:12:53.775036 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:53.774976 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-5d696" event={"ID":"7093efec-8b60-4b8f-821d-42e3c2b4fbe6","Type":"ContainerDied","Data":"63e9c7d8f61d811a7bbcf71a482f5ade0c1cbf2574e78f1bb9887202e04890b5"} Jul 08 19:12:53.775036 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:53.774998 2558 scope.go:117] "RemoveContainer" containerID="10d4af923735d013cd80d0bc30439e7f37363f3d711d915c38b0dfb7fc883ee1" Jul 08 19:12:53.783630 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:53.783611 2558 scope.go:117] "RemoveContainer" containerID="10d4af923735d013cd80d0bc30439e7f37363f3d711d915c38b0dfb7fc883ee1" Jul 08 19:12:53.783881 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:12:53.783862 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10d4af923735d013cd80d0bc30439e7f37363f3d711d915c38b0dfb7fc883ee1\": container with ID starting with 10d4af923735d013cd80d0bc30439e7f37363f3d711d915c38b0dfb7fc883ee1 not found: ID does not exist" containerID="10d4af923735d013cd80d0bc30439e7f37363f3d711d915c38b0dfb7fc883ee1" Jul 08 19:12:53.783937 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:53.783890 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10d4af923735d013cd80d0bc30439e7f37363f3d711d915c38b0dfb7fc883ee1"} err="failed to get container status \"10d4af923735d013cd80d0bc30439e7f37363f3d711d915c38b0dfb7fc883ee1\": rpc error: code = NotFound desc = could not find container \"10d4af923735d013cd80d0bc30439e7f37363f3d711d915c38b0dfb7fc883ee1\": container with ID starting with 10d4af923735d013cd80d0bc30439e7f37363f3d711d915c38b0dfb7fc883ee1 not found: ID does not exist" Jul 08 19:12:53.795644 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:53.795622 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5d696"] Jul 08 19:12:53.799672 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:53.799647 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5d696"] Jul 08 19:12:55.281373 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:55.281327 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7093efec-8b60-4b8f-821d-42e3c2b4fbe6" path="/var/lib/kubelet/pods/7093efec-8b60-4b8f-821d-42e3c2b4fbe6/volumes" Jul 08 19:12:58.653865 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:58.653823 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:12:58.803917 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:58.803883 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:12:59.242745 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:59.242707 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-vlb8h"] Jul 08 19:12:59.243049 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:59.243036 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7093efec-8b60-4b8f-821d-42e3c2b4fbe6" containerName="limitador" Jul 08 19:12:59.243093 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:59.243051 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7093efec-8b60-4b8f-821d-42e3c2b4fbe6" containerName="limitador" Jul 08 19:12:59.243127 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:59.243110 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7093efec-8b60-4b8f-821d-42e3c2b4fbe6" containerName="limitador" Jul 08 19:12:59.248780 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:59.248761 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vlb8h" Jul 08 19:12:59.252185 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:59.252160 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 08 19:12:59.253939 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:59.253918 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-w7929\"" Jul 08 19:12:59.253939 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:59.253931 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 08 19:12:59.254106 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:59.253932 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 08 19:12:59.308681 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:59.308632 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp86h\" (UniqueName: \"kubernetes.io/projected/a4686e35-4b92-4f06-add9-e2feb0ec4fae-kube-api-access-lp86h\") pod \"postgres-69fb4594cf-vlb8h\" (UID: \"a4686e35-4b92-4f06-add9-e2feb0ec4fae\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vlb8h" Jul 08 19:12:59.308681 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:59.308681 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a4686e35-4b92-4f06-add9-e2feb0ec4fae-data\") pod \"postgres-69fb4594cf-vlb8h\" (UID: \"a4686e35-4b92-4f06-add9-e2feb0ec4fae\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vlb8h" Jul 08 19:12:59.311216 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:59.311195 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-vlb8h"] Jul 08 19:12:59.320173 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:59.320145 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:12:59.409266 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:59.409229 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp86h\" (UniqueName: \"kubernetes.io/projected/a4686e35-4b92-4f06-add9-e2feb0ec4fae-kube-api-access-lp86h\") pod \"postgres-69fb4594cf-vlb8h\" (UID: \"a4686e35-4b92-4f06-add9-e2feb0ec4fae\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vlb8h" Jul 08 19:12:59.409426 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:59.409277 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a4686e35-4b92-4f06-add9-e2feb0ec4fae-data\") pod \"postgres-69fb4594cf-vlb8h\" (UID: \"a4686e35-4b92-4f06-add9-e2feb0ec4fae\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vlb8h" Jul 08 19:12:59.409715 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:59.409691 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a4686e35-4b92-4f06-add9-e2feb0ec4fae-data\") pod \"postgres-69fb4594cf-vlb8h\" (UID: \"a4686e35-4b92-4f06-add9-e2feb0ec4fae\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vlb8h" Jul 08 19:12:59.422844 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:59.422825 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp86h\" (UniqueName: \"kubernetes.io/projected/a4686e35-4b92-4f06-add9-e2feb0ec4fae-kube-api-access-lp86h\") pod \"postgres-69fb4594cf-vlb8h\" (UID: \"a4686e35-4b92-4f06-add9-e2feb0ec4fae\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vlb8h" Jul 08 19:12:59.558302 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:59.558209 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vlb8h" Jul 08 19:12:59.702573 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:59.702550 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-vlb8h"] Jul 08 19:12:59.704651 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:12:59.704626 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4686e35_4b92_4f06_add9_e2feb0ec4fae.slice/crio-418de1a9da11ffeb7d1d72b4e09026553137f13b6f8079d48a516ac081a81ad3 WatchSource:0}: Error finding container 418de1a9da11ffeb7d1d72b4e09026553137f13b6f8079d48a516ac081a81ad3: Status 404 returned error can't find the container with id 418de1a9da11ffeb7d1d72b4e09026553137f13b6f8079d48a516ac081a81ad3 Jul 08 19:12:59.807824 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:12:59.807795 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vlb8h" event={"ID":"a4686e35-4b92-4f06-add9-e2feb0ec4fae","Type":"ContainerStarted","Data":"418de1a9da11ffeb7d1d72b4e09026553137f13b6f8079d48a516ac081a81ad3"} Jul 08 19:13:04.838953 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:04.838912 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vlb8h" event={"ID":"a4686e35-4b92-4f06-add9-e2feb0ec4fae","Type":"ContainerStarted","Data":"11178547f55ce028a2f3f29c12cee71c3ab76992a7fb832e3d34bf69afd5333e"} Jul 08 19:13:04.839326 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:04.838978 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vlb8h" Jul 08 19:13:04.868045 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:04.867995 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vlb8h" podStartSLOduration=1.269996909 podStartE2EDuration="5.867979583s" totalImagesPullingTime="4.597982674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:12:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:13:04.866329822 +0000 UTC m=+568.125282764" watchObservedRunningTime="2026-07-08 19:13:04.867979583 +0000 UTC m=+568.126932930" Jul 08 19:13:10.873664 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:10.873636 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vlb8h" Jul 08 19:13:11.211645 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:11.211501 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:13:11.756803 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:11.756766 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:13:18.203376 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:18.203343 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:13:18.244193 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:18.244165 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5975f58b54-lwss9"] Jul 08 19:13:18.247415 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:18.247399 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5975f58b54-lwss9" Jul 08 19:13:18.263730 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:18.263709 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-92bhq\"" Jul 08 19:13:18.263730 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:18.263725 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 08 19:13:18.263877 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:18.263709 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 08 19:13:18.285243 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:18.285214 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5975f58b54-lwss9"] Jul 08 19:13:18.328553 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:18.328519 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:13:18.370646 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:18.370620 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3925c239-835f-4b31-ac1a-a0150381328c-webhook-cert\") pod \"maas-controller-5975f58b54-lwss9\" (UID: \"3925c239-835f-4b31-ac1a-a0150381328c\") " pod="opendatahub/maas-controller-5975f58b54-lwss9" Jul 08 19:13:18.370839 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:18.370817 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d62mf\" (UniqueName: \"kubernetes.io/projected/3925c239-835f-4b31-ac1a-a0150381328c-kube-api-access-d62mf\") pod \"maas-controller-5975f58b54-lwss9\" (UID: \"3925c239-835f-4b31-ac1a-a0150381328c\") " pod="opendatahub/maas-controller-5975f58b54-lwss9" Jul 08 19:13:18.436491 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:18.436462 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:13:18.471438 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:18.471362 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d62mf\" (UniqueName: \"kubernetes.io/projected/3925c239-835f-4b31-ac1a-a0150381328c-kube-api-access-d62mf\") pod \"maas-controller-5975f58b54-lwss9\" (UID: \"3925c239-835f-4b31-ac1a-a0150381328c\") " pod="opendatahub/maas-controller-5975f58b54-lwss9" Jul 08 19:13:18.471438 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:18.471407 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3925c239-835f-4b31-ac1a-a0150381328c-webhook-cert\") pod \"maas-controller-5975f58b54-lwss9\" (UID: \"3925c239-835f-4b31-ac1a-a0150381328c\") " pod="opendatahub/maas-controller-5975f58b54-lwss9" Jul 08 19:13:18.472264 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:18.472240 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-796d55d59-4t5zn"] Jul 08 19:13:18.473790 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:18.473769 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3925c239-835f-4b31-ac1a-a0150381328c-webhook-cert\") pod \"maas-controller-5975f58b54-lwss9\" (UID: \"3925c239-835f-4b31-ac1a-a0150381328c\") " pod="opendatahub/maas-controller-5975f58b54-lwss9" Jul 08 19:13:18.475502 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:18.475485 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-796d55d59-4t5zn" Jul 08 19:13:18.510504 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:18.510468 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-796d55d59-4t5zn"] Jul 08 19:13:18.524812 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:18.524776 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d62mf\" (UniqueName: \"kubernetes.io/projected/3925c239-835f-4b31-ac1a-a0150381328c-kube-api-access-d62mf\") pod \"maas-controller-5975f58b54-lwss9\" (UID: \"3925c239-835f-4b31-ac1a-a0150381328c\") " pod="opendatahub/maas-controller-5975f58b54-lwss9" Jul 08 19:13:18.557543 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:18.557511 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5975f58b54-lwss9" Jul 08 19:13:18.572070 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:18.572035 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e978acd9-810b-4760-a61e-d3ad50573633-webhook-cert\") pod \"maas-controller-796d55d59-4t5zn\" (UID: \"e978acd9-810b-4760-a61e-d3ad50573633\") " pod="opendatahub/maas-controller-796d55d59-4t5zn" Jul 08 19:13:18.572199 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:18.572159 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-595bj\" (UniqueName: \"kubernetes.io/projected/e978acd9-810b-4760-a61e-d3ad50573633-kube-api-access-595bj\") pod \"maas-controller-796d55d59-4t5zn\" (UID: \"e978acd9-810b-4760-a61e-d3ad50573633\") " pod="opendatahub/maas-controller-796d55d59-4t5zn" Jul 08 19:13:18.673293 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:18.673262 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e978acd9-810b-4760-a61e-d3ad50573633-webhook-cert\") pod \"maas-controller-796d55d59-4t5zn\" (UID: \"e978acd9-810b-4760-a61e-d3ad50573633\") " pod="opendatahub/maas-controller-796d55d59-4t5zn" Jul 08 19:13:18.673443 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:18.673325 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-595bj\" (UniqueName: \"kubernetes.io/projected/e978acd9-810b-4760-a61e-d3ad50573633-kube-api-access-595bj\") pod \"maas-controller-796d55d59-4t5zn\" (UID: \"e978acd9-810b-4760-a61e-d3ad50573633\") " pod="opendatahub/maas-controller-796d55d59-4t5zn" Jul 08 19:13:18.675830 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:18.675804 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e978acd9-810b-4760-a61e-d3ad50573633-webhook-cert\") pod \"maas-controller-796d55d59-4t5zn\" (UID: \"e978acd9-810b-4760-a61e-d3ad50573633\") " pod="opendatahub/maas-controller-796d55d59-4t5zn" Jul 08 19:13:18.684136 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:18.684114 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-595bj\" (UniqueName: \"kubernetes.io/projected/e978acd9-810b-4760-a61e-d3ad50573633-kube-api-access-595bj\") pod \"maas-controller-796d55d59-4t5zn\" (UID: \"e978acd9-810b-4760-a61e-d3ad50573633\") " pod="opendatahub/maas-controller-796d55d59-4t5zn" Jul 08 19:13:18.789913 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:18.789827 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-796d55d59-4t5zn" Jul 08 19:13:18.910521 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:18.910390 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5975f58b54-lwss9"] Jul 08 19:13:18.913750 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:13:18.913720 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3925c239_835f_4b31_ac1a_a0150381328c.slice/crio-d3a27919e2d62dbf7e2aeee8ad6e916fb0b905dec2ae54a858515fd117ce0609 WatchSource:0}: Error finding container d3a27919e2d62dbf7e2aeee8ad6e916fb0b905dec2ae54a858515fd117ce0609: Status 404 returned error can't find the container with id d3a27919e2d62dbf7e2aeee8ad6e916fb0b905dec2ae54a858515fd117ce0609 Jul 08 19:13:18.948295 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:18.948271 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-796d55d59-4t5zn"] Jul 08 19:13:18.950395 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:13:18.950370 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode978acd9_810b_4760_a61e_d3ad50573633.slice/crio-ea16794a61d76cc69f296ba37af7e19c94c7ca09e2922799382d8a2716e5e9fb WatchSource:0}: Error finding container ea16794a61d76cc69f296ba37af7e19c94c7ca09e2922799382d8a2716e5e9fb: Status 404 returned error can't find the container with id ea16794a61d76cc69f296ba37af7e19c94c7ca09e2922799382d8a2716e5e9fb Jul 08 19:13:19.919885 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:19.919847 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5975f58b54-lwss9" event={"ID":"3925c239-835f-4b31-ac1a-a0150381328c","Type":"ContainerStarted","Data":"d3a27919e2d62dbf7e2aeee8ad6e916fb0b905dec2ae54a858515fd117ce0609"} Jul 08 19:13:19.921403 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:19.921354 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-796d55d59-4t5zn" event={"ID":"e978acd9-810b-4760-a61e-d3ad50573633","Type":"ContainerStarted","Data":"ea16794a61d76cc69f296ba37af7e19c94c7ca09e2922799382d8a2716e5e9fb"} Jul 08 19:13:21.324411 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:21.324366 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:13:21.420324 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:21.420292 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:13:21.935233 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:21.935191 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-796d55d59-4t5zn" event={"ID":"e978acd9-810b-4760-a61e-d3ad50573633","Type":"ContainerStarted","Data":"938a9b4c5393e6b5608c256a9f2d1ac608f2540d514ffcd951d25d30ddb26b92"} Jul 08 19:13:21.935461 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:21.935366 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-796d55d59-4t5zn" Jul 08 19:13:21.936740 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:21.936714 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5975f58b54-lwss9" event={"ID":"3925c239-835f-4b31-ac1a-a0150381328c","Type":"ContainerStarted","Data":"d1537310f5818a99ab9a9614365975502aaf93a7c54e21019a43eeab0bd8c8d4"} Jul 08 19:13:21.936851 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:21.936774 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5975f58b54-lwss9" Jul 08 19:13:21.997799 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:21.997754 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-796d55d59-4t5zn" podStartSLOduration=1.847795639 podStartE2EDuration="3.997740573s" totalImagesPullingTime="2.149944934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:13:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:13:21.967114332 +0000 UTC m=+585.226067259" watchObservedRunningTime="2026-07-08 19:13:21.997740573 +0000 UTC m=+585.256693512" Jul 08 19:13:21.998055 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:21.998035 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5975f58b54-lwss9" podStartSLOduration=1.8214495560000001 podStartE2EDuration="3.99802965s" totalImagesPullingTime="2.176580094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:13:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:13:21.995344505 +0000 UTC m=+585.254297430" watchObservedRunningTime="2026-07-08 19:13:21.99802965 +0000 UTC m=+585.256982574" Jul 08 19:13:27.588020 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:27.587989 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:13:27.693223 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:27.693192 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:13:32.949895 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:32.949863 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-796d55d59-4t5zn" Jul 08 19:13:32.950279 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:32.949930 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5975f58b54-lwss9" Jul 08 19:13:33.026218 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:33.026185 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5975f58b54-lwss9"] Jul 08 19:13:33.026429 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:33.026404 2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-5975f58b54-lwss9" podUID="3925c239-835f-4b31-ac1a-a0150381328c" containerName="manager" containerID="cri-o://d1537310f5818a99ab9a9614365975502aaf93a7c54e21019a43eeab0bd8c8d4" gracePeriod=30 Jul 08 19:13:33.263618 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:33.263572 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5975f58b54-lwss9" Jul 08 19:13:33.402905 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:33.402872 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3925c239-835f-4b31-ac1a-a0150381328c-webhook-cert\") pod \"3925c239-835f-4b31-ac1a-a0150381328c\" (UID: \"3925c239-835f-4b31-ac1a-a0150381328c\") " Jul 08 19:13:33.403083 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:33.402941 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d62mf\" (UniqueName: \"kubernetes.io/projected/3925c239-835f-4b31-ac1a-a0150381328c-kube-api-access-d62mf\") pod \"3925c239-835f-4b31-ac1a-a0150381328c\" (UID: \"3925c239-835f-4b31-ac1a-a0150381328c\") " Jul 08 19:13:33.405136 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:33.405099 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3925c239-835f-4b31-ac1a-a0150381328c-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "3925c239-835f-4b31-ac1a-a0150381328c" (UID: "3925c239-835f-4b31-ac1a-a0150381328c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:13:33.405136 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:33.405099 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3925c239-835f-4b31-ac1a-a0150381328c-kube-api-access-d62mf" (OuterVolumeSpecName: "kube-api-access-d62mf") pod "3925c239-835f-4b31-ac1a-a0150381328c" (UID: "3925c239-835f-4b31-ac1a-a0150381328c"). InnerVolumeSpecName "kube-api-access-d62mf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:13:33.503868 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:33.503776 2558 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3925c239-835f-4b31-ac1a-a0150381328c-webhook-cert\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:13:33.503868 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:33.503817 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d62mf\" (UniqueName: \"kubernetes.io/projected/3925c239-835f-4b31-ac1a-a0150381328c-kube-api-access-d62mf\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:13:33.995526 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:33.995495 2558 generic.go:356] "Generic (PLEG): container finished" podID="3925c239-835f-4b31-ac1a-a0150381328c" containerID="d1537310f5818a99ab9a9614365975502aaf93a7c54e21019a43eeab0bd8c8d4" exitCode=0 Jul 08 19:13:33.995934 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:33.995538 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5975f58b54-lwss9" Jul 08 19:13:33.995934 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:33.995603 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5975f58b54-lwss9" event={"ID":"3925c239-835f-4b31-ac1a-a0150381328c","Type":"ContainerDied","Data":"d1537310f5818a99ab9a9614365975502aaf93a7c54e21019a43eeab0bd8c8d4"} Jul 08 19:13:33.995934 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:33.995660 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5975f58b54-lwss9" event={"ID":"3925c239-835f-4b31-ac1a-a0150381328c","Type":"ContainerDied","Data":"d3a27919e2d62dbf7e2aeee8ad6e916fb0b905dec2ae54a858515fd117ce0609"} Jul 08 19:13:33.995934 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:33.995684 2558 scope.go:117] "RemoveContainer" containerID="d1537310f5818a99ab9a9614365975502aaf93a7c54e21019a43eeab0bd8c8d4" Jul 08 19:13:34.004674 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:34.004656 2558 scope.go:117] "RemoveContainer" containerID="d1537310f5818a99ab9a9614365975502aaf93a7c54e21019a43eeab0bd8c8d4" Jul 08 19:13:34.004915 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:13:34.004890 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1537310f5818a99ab9a9614365975502aaf93a7c54e21019a43eeab0bd8c8d4\": container with ID starting with d1537310f5818a99ab9a9614365975502aaf93a7c54e21019a43eeab0bd8c8d4 not found: ID does not exist" containerID="d1537310f5818a99ab9a9614365975502aaf93a7c54e21019a43eeab0bd8c8d4" Jul 08 19:13:34.004961 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:34.004918 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1537310f5818a99ab9a9614365975502aaf93a7c54e21019a43eeab0bd8c8d4"} err="failed to get container status \"d1537310f5818a99ab9a9614365975502aaf93a7c54e21019a43eeab0bd8c8d4\": rpc error: code = NotFound desc = could not find container \"d1537310f5818a99ab9a9614365975502aaf93a7c54e21019a43eeab0bd8c8d4\": container with ID starting with d1537310f5818a99ab9a9614365975502aaf93a7c54e21019a43eeab0bd8c8d4 not found: ID does not exist" Jul 08 19:13:34.034449 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:34.034424 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5975f58b54-lwss9"] Jul 08 19:13:34.045073 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:34.045051 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5975f58b54-lwss9"] Jul 08 19:13:35.281765 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:35.281735 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3925c239-835f-4b31-ac1a-a0150381328c" path="/var/lib/kubelet/pods/3925c239-835f-4b31-ac1a-a0150381328c/volumes" Jul 08 19:13:37.174889 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:37.174860 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-54hnr_1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a/console-operator/2.log" Jul 08 19:13:37.176713 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:37.176689 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-54hnr_1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a/console-operator/2.log" Jul 08 19:13:53.255903 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:53.255833 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:13:53.815936 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:53.815903 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:13:53.820556 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:53.820534 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-78b9fc9ff8-rfkzz"] Jul 08 19:13:53.820876 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:53.820864 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3925c239-835f-4b31-ac1a-a0150381328c" containerName="manager" Jul 08 19:13:53.820922 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:53.820878 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3925c239-835f-4b31-ac1a-a0150381328c" containerName="manager" Jul 08 19:13:53.820955 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:53.820929 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3925c239-835f-4b31-ac1a-a0150381328c" containerName="manager" Jul 08 19:13:53.822783 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:53.822765 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78b9fc9ff8-rfkzz" Jul 08 19:13:53.825278 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:53.825254 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 08 19:13:53.825278 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:53.825271 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8lfvd\"" Jul 08 19:13:53.835038 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:53.835018 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-78b9fc9ff8-rfkzz"] Jul 08 19:13:53.866741 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:53.866715 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swbhg\" (UniqueName: \"kubernetes.io/projected/6455707e-dbb0-4815-b8d9-d3e41f42da07-kube-api-access-swbhg\") pod \"maas-api-78b9fc9ff8-rfkzz\" (UID: \"6455707e-dbb0-4815-b8d9-d3e41f42da07\") " pod="odh-ai-gateway-infra/maas-api-78b9fc9ff8-rfkzz" Jul 08 19:13:53.866845 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:53.866755 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6455707e-dbb0-4815-b8d9-d3e41f42da07-maas-api-tls\") pod \"maas-api-78b9fc9ff8-rfkzz\" (UID: \"6455707e-dbb0-4815-b8d9-d3e41f42da07\") " pod="odh-ai-gateway-infra/maas-api-78b9fc9ff8-rfkzz" Jul 08 19:13:53.866845 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:53.866824 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6455707e-dbb0-4815-b8d9-d3e41f42da07-openshift-service-ca\") pod \"maas-api-78b9fc9ff8-rfkzz\" (UID: \"6455707e-dbb0-4815-b8d9-d3e41f42da07\") " pod="odh-ai-gateway-infra/maas-api-78b9fc9ff8-rfkzz" Jul 08 19:13:53.967800 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:53.967766 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6455707e-dbb0-4815-b8d9-d3e41f42da07-openshift-service-ca\") pod \"maas-api-78b9fc9ff8-rfkzz\" (UID: \"6455707e-dbb0-4815-b8d9-d3e41f42da07\") " pod="odh-ai-gateway-infra/maas-api-78b9fc9ff8-rfkzz" Jul 08 19:13:53.967989 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:53.967814 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swbhg\" (UniqueName: \"kubernetes.io/projected/6455707e-dbb0-4815-b8d9-d3e41f42da07-kube-api-access-swbhg\") pod \"maas-api-78b9fc9ff8-rfkzz\" (UID: \"6455707e-dbb0-4815-b8d9-d3e41f42da07\") " pod="odh-ai-gateway-infra/maas-api-78b9fc9ff8-rfkzz" Jul 08 19:13:53.967989 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:53.967836 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6455707e-dbb0-4815-b8d9-d3e41f42da07-maas-api-tls\") pod \"maas-api-78b9fc9ff8-rfkzz\" (UID: \"6455707e-dbb0-4815-b8d9-d3e41f42da07\") " pod="odh-ai-gateway-infra/maas-api-78b9fc9ff8-rfkzz" Jul 08 19:13:53.967989 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:13:53.967946 2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 08 19:13:53.968153 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:13:53.968009 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6455707e-dbb0-4815-b8d9-d3e41f42da07-maas-api-tls podName:6455707e-dbb0-4815-b8d9-d3e41f42da07 nodeName:}" failed. No retries permitted until 2026-07-08 19:13:54.46798869 +0000 UTC m=+617.726941594 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6455707e-dbb0-4815-b8d9-d3e41f42da07-maas-api-tls") pod "maas-api-78b9fc9ff8-rfkzz" (UID: "6455707e-dbb0-4815-b8d9-d3e41f42da07") : secret "maas-api-serving-cert" not found Jul 08 19:13:53.968529 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:53.968508 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6455707e-dbb0-4815-b8d9-d3e41f42da07-openshift-service-ca\") pod \"maas-api-78b9fc9ff8-rfkzz\" (UID: \"6455707e-dbb0-4815-b8d9-d3e41f42da07\") " pod="odh-ai-gateway-infra/maas-api-78b9fc9ff8-rfkzz" Jul 08 19:13:53.977306 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:53.977281 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swbhg\" (UniqueName: \"kubernetes.io/projected/6455707e-dbb0-4815-b8d9-d3e41f42da07-kube-api-access-swbhg\") pod \"maas-api-78b9fc9ff8-rfkzz\" (UID: \"6455707e-dbb0-4815-b8d9-d3e41f42da07\") " pod="odh-ai-gateway-infra/maas-api-78b9fc9ff8-rfkzz" Jul 08 19:13:54.095883 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:54.095778 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:13:54.234518 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:54.234483 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:13:54.334877 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:54.334843 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:13:54.380367 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:54.380337 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-7cj9f"] Jul 08 19:13:54.382716 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:54.382700 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-7cj9f" Jul 08 19:13:54.385320 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:54.385297 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-gtd75\"" Jul 08 19:13:54.385320 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:54.385297 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 08 19:13:54.394762 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:54.394739 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-7cj9f"] Jul 08 19:13:54.429854 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:54.429833 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-vtjcn"] Jul 08 19:13:54.432652 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:54.432636 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-vtjcn" Jul 08 19:13:54.441763 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:54.441738 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-vtjcn"] Jul 08 19:13:54.471887 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:54.471855 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghg9m\" (UniqueName: \"kubernetes.io/projected/a41a36b0-53e5-4928-9147-03e60bf26b42-kube-api-access-ghg9m\") pod \"payload-processing-58f86c7798-7cj9f\" (UID: \"a41a36b0-53e5-4928-9147-03e60bf26b42\") " pod="openshift-ingress/payload-processing-58f86c7798-7cj9f" Jul 08 19:13:54.472027 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:54.471890 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/434297f0-bb7f-4957-adcb-101410815ed8-plugins-config-volume\") pod \"payload-pre-processing-7964457696-vtjcn\" (UID: \"434297f0-bb7f-4957-adcb-101410815ed8\") " pod="openshift-ingress/payload-pre-processing-7964457696-vtjcn" Jul 08 19:13:54.472027 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:54.471920 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6455707e-dbb0-4815-b8d9-d3e41f42da07-maas-api-tls\") pod \"maas-api-78b9fc9ff8-rfkzz\" (UID: \"6455707e-dbb0-4815-b8d9-d3e41f42da07\") " pod="odh-ai-gateway-infra/maas-api-78b9fc9ff8-rfkzz" Jul 08 19:13:54.472027 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:54.471966 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w8f8\" (UniqueName: \"kubernetes.io/projected/434297f0-bb7f-4957-adcb-101410815ed8-kube-api-access-8w8f8\") pod \"payload-pre-processing-7964457696-vtjcn\" (UID: \"434297f0-bb7f-4957-adcb-101410815ed8\") " pod="openshift-ingress/payload-pre-processing-7964457696-vtjcn" Jul 08 19:13:54.472239 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:54.472044 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a41a36b0-53e5-4928-9147-03e60bf26b42-plugins-config-volume\") pod \"payload-processing-58f86c7798-7cj9f\" (UID: \"a41a36b0-53e5-4928-9147-03e60bf26b42\") " pod="openshift-ingress/payload-processing-58f86c7798-7cj9f" Jul 08 19:13:54.474377 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:54.474355 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6455707e-dbb0-4815-b8d9-d3e41f42da07-maas-api-tls\") pod \"maas-api-78b9fc9ff8-rfkzz\" (UID: \"6455707e-dbb0-4815-b8d9-d3e41f42da07\") " pod="odh-ai-gateway-infra/maas-api-78b9fc9ff8-rfkzz" Jul 08 19:13:54.573008 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:54.572961 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w8f8\" (UniqueName: \"kubernetes.io/projected/434297f0-bb7f-4957-adcb-101410815ed8-kube-api-access-8w8f8\") pod \"payload-pre-processing-7964457696-vtjcn\" (UID: \"434297f0-bb7f-4957-adcb-101410815ed8\") " pod="openshift-ingress/payload-pre-processing-7964457696-vtjcn" Jul 08 19:13:54.573008 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:54.573007 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a41a36b0-53e5-4928-9147-03e60bf26b42-plugins-config-volume\") pod \"payload-processing-58f86c7798-7cj9f\" (UID: \"a41a36b0-53e5-4928-9147-03e60bf26b42\") " pod="openshift-ingress/payload-processing-58f86c7798-7cj9f" Jul 08 19:13:54.573273 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:54.573072 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghg9m\" (UniqueName: \"kubernetes.io/projected/a41a36b0-53e5-4928-9147-03e60bf26b42-kube-api-access-ghg9m\") pod \"payload-processing-58f86c7798-7cj9f\" (UID: \"a41a36b0-53e5-4928-9147-03e60bf26b42\") " pod="openshift-ingress/payload-processing-58f86c7798-7cj9f" Jul 08 19:13:54.573273 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:54.573106 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/434297f0-bb7f-4957-adcb-101410815ed8-plugins-config-volume\") pod \"payload-pre-processing-7964457696-vtjcn\" (UID: \"434297f0-bb7f-4957-adcb-101410815ed8\") " pod="openshift-ingress/payload-pre-processing-7964457696-vtjcn" Jul 08 19:13:54.574075 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:54.574044 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a41a36b0-53e5-4928-9147-03e60bf26b42-plugins-config-volume\") pod \"payload-processing-58f86c7798-7cj9f\" (UID: \"a41a36b0-53e5-4928-9147-03e60bf26b42\") " pod="openshift-ingress/payload-processing-58f86c7798-7cj9f" Jul 08 19:13:54.574229 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:54.574044 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/434297f0-bb7f-4957-adcb-101410815ed8-plugins-config-volume\") pod \"payload-pre-processing-7964457696-vtjcn\" (UID: \"434297f0-bb7f-4957-adcb-101410815ed8\") " pod="openshift-ingress/payload-pre-processing-7964457696-vtjcn" Jul 08 19:13:54.586363 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:54.586331 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w8f8\" (UniqueName: \"kubernetes.io/projected/434297f0-bb7f-4957-adcb-101410815ed8-kube-api-access-8w8f8\") pod \"payload-pre-processing-7964457696-vtjcn\" (UID: \"434297f0-bb7f-4957-adcb-101410815ed8\") " pod="openshift-ingress/payload-pre-processing-7964457696-vtjcn" Jul 08 19:13:54.586490 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:54.586389 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghg9m\" (UniqueName: \"kubernetes.io/projected/a41a36b0-53e5-4928-9147-03e60bf26b42-kube-api-access-ghg9m\") pod \"payload-processing-58f86c7798-7cj9f\" (UID: \"a41a36b0-53e5-4928-9147-03e60bf26b42\") " pod="openshift-ingress/payload-processing-58f86c7798-7cj9f" Jul 08 19:13:54.694611 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:54.694519 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-7cj9f" Jul 08 19:13:54.732918 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:54.732888 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78b9fc9ff8-rfkzz" Jul 08 19:13:54.742673 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:54.742641 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-vtjcn" Jul 08 19:13:54.857028 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:54.856992 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-7cj9f"] Jul 08 19:13:54.859270 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:13:54.859241 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda41a36b0_53e5_4928_9147_03e60bf26b42.slice/crio-71cf5d96eb1ee62e134c5b3dd7a317a0632c2ed0c3f64a3107b39500452fe85f WatchSource:0}: Error finding container 71cf5d96eb1ee62e134c5b3dd7a317a0632c2ed0c3f64a3107b39500452fe85f: Status 404 returned error can't find the container with id 71cf5d96eb1ee62e134c5b3dd7a317a0632c2ed0c3f64a3107b39500452fe85f Jul 08 19:13:54.861297 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:54.861279 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 19:13:54.907474 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:54.907436 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-78b9fc9ff8-rfkzz"] Jul 08 19:13:54.909453 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:13:54.909420 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6455707e_dbb0_4815_b8d9_d3e41f42da07.slice/crio-ee7b46fb373cc0a09c1f3ade1c7773acf1a4b6f3cf554a1910adab63cec5c2e9 WatchSource:0}: Error finding container ee7b46fb373cc0a09c1f3ade1c7773acf1a4b6f3cf554a1910adab63cec5c2e9: Status 404 returned error can't find the container with id ee7b46fb373cc0a09c1f3ade1c7773acf1a4b6f3cf554a1910adab63cec5c2e9 Jul 08 19:13:54.924371 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:54.924335 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-vtjcn"] Jul 08 19:13:54.926764 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:13:54.926740 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod434297f0_bb7f_4957_adcb_101410815ed8.slice/crio-6cccf867bcf36ceeab2bb8cc2168e15b8a613312212b646b904f0eb92b1e0574 WatchSource:0}: Error finding container 6cccf867bcf36ceeab2bb8cc2168e15b8a613312212b646b904f0eb92b1e0574: Status 404 returned error can't find the container with id 6cccf867bcf36ceeab2bb8cc2168e15b8a613312212b646b904f0eb92b1e0574 Jul 08 19:13:55.101909 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:55.101818 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78b9fc9ff8-rfkzz" event={"ID":"6455707e-dbb0-4815-b8d9-d3e41f42da07","Type":"ContainerStarted","Data":"ee7b46fb373cc0a09c1f3ade1c7773acf1a4b6f3cf554a1910adab63cec5c2e9"} Jul 08 19:13:55.102925 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:55.102897 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-vtjcn" event={"ID":"434297f0-bb7f-4957-adcb-101410815ed8","Type":"ContainerStarted","Data":"6cccf867bcf36ceeab2bb8cc2168e15b8a613312212b646b904f0eb92b1e0574"} Jul 08 19:13:55.104058 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:55.104037 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-7cj9f" event={"ID":"a41a36b0-53e5-4928-9147-03e60bf26b42","Type":"ContainerStarted","Data":"71cf5d96eb1ee62e134c5b3dd7a317a0632c2ed0c3f64a3107b39500452fe85f"} Jul 08 19:13:57.117018 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:57.116930 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-7cj9f" event={"ID":"a41a36b0-53e5-4928-9147-03e60bf26b42","Type":"ContainerStarted","Data":"46a88244a95421b7954b1bd96a800f3a6de3f94acc649286918234062f1fb353"} Jul 08 19:13:57.117421 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:57.117064 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-7cj9f" Jul 08 19:13:57.118485 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:57.118461 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78b9fc9ff8-rfkzz" event={"ID":"6455707e-dbb0-4815-b8d9-d3e41f42da07","Type":"ContainerStarted","Data":"18b9574a04fb2e8ad22ace63514ae005c6cde146a2869bf9d6f49a9a2f61e9e0"} Jul 08 19:13:57.118652 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:57.118519 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-78b9fc9ff8-rfkzz" Jul 08 19:13:57.119971 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:57.119939 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-vtjcn" event={"ID":"434297f0-bb7f-4957-adcb-101410815ed8","Type":"ContainerStarted","Data":"2418984d135beaf7f690f16393a2269cfb6187676162e418975f04a60d2dfc2b"} Jul 08 19:13:57.120075 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:57.120014 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-vtjcn" Jul 08 19:13:57.150933 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:57.150892 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-7cj9f" podStartSLOduration=1.336374746 podStartE2EDuration="3.15087896s" totalImagesPullingTime="1.814504214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:13:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:13:57.149890472 +0000 UTC m=+620.408843398" watchObservedRunningTime="2026-07-08 19:13:57.15087896 +0000 UTC m=+620.409831885" Jul 08 19:13:57.221049 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:57.221003 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-vtjcn" podStartSLOduration=1.471255385 podStartE2EDuration="3.220989637s" totalImagesPullingTime="1.749734252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:13:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:13:57.215183053 +0000 UTC m=+620.474135999" watchObservedRunningTime="2026-07-08 19:13:57.220989637 +0000 UTC m=+620.479942561" Jul 08 19:13:57.221569 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:13:57.221545 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-78b9fc9ff8-rfkzz" podStartSLOduration=2.456402757 podStartE2EDuration="4.221539333s" totalImagesPullingTime="1.765136576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:13:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:13:57.184920071 +0000 UTC m=+620.443873083" watchObservedRunningTime="2026-07-08 19:13:57.221539333 +0000 UTC m=+620.480492257" Jul 08 19:14:02.855745 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:02.855641 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:14:02.920961 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:02.920906 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:14:03.130145 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:03.130121 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-78b9fc9ff8-rfkzz" Jul 08 19:14:04.707678 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:04.707634 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:14:06.059103 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:06.059068 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:14:08.127081 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:08.127054 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-7cj9f" Jul 08 19:14:08.127558 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:08.127213 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-vtjcn" Jul 08 19:14:14.401811 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:14.401769 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl"] Jul 08 19:14:14.404337 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:14.404318 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl" Jul 08 19:14:14.408016 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:14.407997 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-7lzf5\"" Jul 08 19:14:14.408016 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:14.408004 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 08 19:14:14.408193 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:14.407997 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 08 19:14:14.408193 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:14.407998 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 08 19:14:14.414827 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:14.414802 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl"] Jul 08 19:14:14.418141 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:14.418115 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:14:14.548242 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:14.548207 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8fd614e-0542-4c5c-997b-683bca306081-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl\" (UID: \"a8fd614e-0542-4c5c-997b-683bca306081\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl" Jul 08 19:14:14.548242 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:14.548243 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8fd614e-0542-4c5c-997b-683bca306081-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl\" (UID: \"a8fd614e-0542-4c5c-997b-683bca306081\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl" Jul 08 19:14:14.548435 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:14.548267 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8fd614e-0542-4c5c-997b-683bca306081-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl\" (UID: \"a8fd614e-0542-4c5c-997b-683bca306081\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl" Jul 08 19:14:14.548435 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:14.548292 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8fd614e-0542-4c5c-997b-683bca306081-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl\" (UID: \"a8fd614e-0542-4c5c-997b-683bca306081\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl" Jul 08 19:14:14.548435 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:14.548335 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8fd614e-0542-4c5c-997b-683bca306081-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl\" (UID: \"a8fd614e-0542-4c5c-997b-683bca306081\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl" Jul 08 19:14:14.548435 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:14.548364 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k7mt\" (UniqueName: \"kubernetes.io/projected/a8fd614e-0542-4c5c-997b-683bca306081-kube-api-access-9k7mt\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl\" (UID: \"a8fd614e-0542-4c5c-997b-683bca306081\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl" Jul 08 19:14:14.649586 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:14.649553 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8fd614e-0542-4c5c-997b-683bca306081-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl\" (UID: \"a8fd614e-0542-4c5c-997b-683bca306081\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl" Jul 08 19:14:14.649744 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:14.649612 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k7mt\" (UniqueName: \"kubernetes.io/projected/a8fd614e-0542-4c5c-997b-683bca306081-kube-api-access-9k7mt\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl\" (UID: \"a8fd614e-0542-4c5c-997b-683bca306081\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl" Jul 08 19:14:14.649744 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:14.649661 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8fd614e-0542-4c5c-997b-683bca306081-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl\" (UID: \"a8fd614e-0542-4c5c-997b-683bca306081\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl" Jul 08 19:14:14.649744 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:14.649677 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8fd614e-0542-4c5c-997b-683bca306081-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl\" (UID: \"a8fd614e-0542-4c5c-997b-683bca306081\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl" Jul 08 19:14:14.649744 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:14.649702 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8fd614e-0542-4c5c-997b-683bca306081-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl\" (UID: \"a8fd614e-0542-4c5c-997b-683bca306081\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl" Jul 08 19:14:14.649744 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:14.649726 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8fd614e-0542-4c5c-997b-683bca306081-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl\" (UID: \"a8fd614e-0542-4c5c-997b-683bca306081\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl" Jul 08 19:14:14.650026 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:14.650002 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8fd614e-0542-4c5c-997b-683bca306081-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl\" (UID: \"a8fd614e-0542-4c5c-997b-683bca306081\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl" Jul 08 19:14:14.650095 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:14.650059 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8fd614e-0542-4c5c-997b-683bca306081-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl\" (UID: \"a8fd614e-0542-4c5c-997b-683bca306081\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl" Jul 08 19:14:14.650189 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:14.650169 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8fd614e-0542-4c5c-997b-683bca306081-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl\" (UID: \"a8fd614e-0542-4c5c-997b-683bca306081\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl" Jul 08 19:14:14.651878 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:14.651828 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8fd614e-0542-4c5c-997b-683bca306081-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl\" (UID: \"a8fd614e-0542-4c5c-997b-683bca306081\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl" Jul 08 19:14:14.652130 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:14.652111 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8fd614e-0542-4c5c-997b-683bca306081-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl\" (UID: \"a8fd614e-0542-4c5c-997b-683bca306081\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl" Jul 08 19:14:14.660063 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:14.660032 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k7mt\" (UniqueName: \"kubernetes.io/projected/a8fd614e-0542-4c5c-997b-683bca306081-kube-api-access-9k7mt\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl\" (UID: \"a8fd614e-0542-4c5c-997b-683bca306081\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl" Jul 08 19:14:14.716995 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:14.716973 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl" Jul 08 19:14:14.851200 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:14.851174 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl"] Jul 08 19:14:14.852864 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:14:14.852842 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8fd614e_0542_4c5c_997b_683bca306081.slice/crio-e807272ca6c829a39f256eb2d109347a416b8752ee3439280b7f1c1e8b9e709a WatchSource:0}: Error finding container e807272ca6c829a39f256eb2d109347a416b8752ee3439280b7f1c1e8b9e709a: Status 404 returned error can't find the container with id e807272ca6c829a39f256eb2d109347a416b8752ee3439280b7f1c1e8b9e709a Jul 08 19:14:15.215060 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:15.215011 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl" event={"ID":"a8fd614e-0542-4c5c-997b-683bca306081","Type":"ContainerStarted","Data":"e807272ca6c829a39f256eb2d109347a416b8752ee3439280b7f1c1e8b9e709a"} Jul 08 19:14:16.221385 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:16.221288 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl" event={"ID":"a8fd614e-0542-4c5c-997b-683bca306081","Type":"ContainerStarted","Data":"27a6b33ee1c74c28adef0b83cb03e8b3661648dfda19da0c15a4b6f24f8a3f89"} Jul 08 19:14:16.241491 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:16.241436 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl" podStartSLOduration=1.172847837 podStartE2EDuration="2.241421442s" totalImagesPullingTime="1.068573605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:14:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:14:16.239088825 +0000 UTC m=+639.498041748" watchObservedRunningTime="2026-07-08 19:14:16.241421442 +0000 UTC m=+639.500374364" Jul 08 19:14:24.718097 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:24.718060 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl" Jul 08 19:14:24.718629 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:24.718118 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl" Jul 08 19:14:24.732892 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:24.732868 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl" Jul 08 19:14:25.282147 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:25.282118 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl" Jul 08 19:14:38.872496 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:38.872459 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:14:44.438324 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:44.438288 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:14:49.312094 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:49.312060 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w"] Jul 08 19:14:49.315493 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:49.315475 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w" Jul 08 19:14:49.320036 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:49.320015 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 08 19:14:49.332518 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:49.332497 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w"] Jul 08 19:14:49.437559 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:49.437529 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c255c6e4-3338-4ded-bb5d-66022aa7d036-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w\" (UID: \"c255c6e4-3338-4ded-bb5d-66022aa7d036\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w" Jul 08 19:14:49.437721 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:49.437569 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c255c6e4-3338-4ded-bb5d-66022aa7d036-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w\" (UID: \"c255c6e4-3338-4ded-bb5d-66022aa7d036\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w" Jul 08 19:14:49.437721 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:49.437669 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76jvc\" (UniqueName: \"kubernetes.io/projected/c255c6e4-3338-4ded-bb5d-66022aa7d036-kube-api-access-76jvc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w\" (UID: \"c255c6e4-3338-4ded-bb5d-66022aa7d036\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w" Jul 08 19:14:49.437721 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:49.437696 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c255c6e4-3338-4ded-bb5d-66022aa7d036-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w\" (UID: \"c255c6e4-3338-4ded-bb5d-66022aa7d036\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w" Jul 08 19:14:49.437721 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:49.437721 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c255c6e4-3338-4ded-bb5d-66022aa7d036-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w\" (UID: \"c255c6e4-3338-4ded-bb5d-66022aa7d036\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w" Jul 08 19:14:49.437884 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:49.437744 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c255c6e4-3338-4ded-bb5d-66022aa7d036-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w\" (UID: \"c255c6e4-3338-4ded-bb5d-66022aa7d036\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w" Jul 08 19:14:49.515220 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:49.515186 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:14:49.539171 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:49.539142 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c255c6e4-3338-4ded-bb5d-66022aa7d036-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w\" (UID: \"c255c6e4-3338-4ded-bb5d-66022aa7d036\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w" Jul 08 19:14:49.539292 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:49.539189 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76jvc\" (UniqueName: \"kubernetes.io/projected/c255c6e4-3338-4ded-bb5d-66022aa7d036-kube-api-access-76jvc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w\" (UID: \"c255c6e4-3338-4ded-bb5d-66022aa7d036\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w" Jul 08 19:14:49.539292 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:49.539207 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c255c6e4-3338-4ded-bb5d-66022aa7d036-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w\" (UID: \"c255c6e4-3338-4ded-bb5d-66022aa7d036\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w" Jul 08 19:14:49.539292 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:49.539227 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c255c6e4-3338-4ded-bb5d-66022aa7d036-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w\" (UID: \"c255c6e4-3338-4ded-bb5d-66022aa7d036\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w" Jul 08 19:14:49.539292 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:49.539247 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c255c6e4-3338-4ded-bb5d-66022aa7d036-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w\" (UID: \"c255c6e4-3338-4ded-bb5d-66022aa7d036\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w" Jul 08 19:14:49.539460 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:49.539296 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c255c6e4-3338-4ded-bb5d-66022aa7d036-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w\" (UID: \"c255c6e4-3338-4ded-bb5d-66022aa7d036\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w" Jul 08 19:14:49.539633 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:49.539612 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c255c6e4-3338-4ded-bb5d-66022aa7d036-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w\" (UID: \"c255c6e4-3338-4ded-bb5d-66022aa7d036\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w" Jul 08 19:14:49.539702 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:49.539638 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c255c6e4-3338-4ded-bb5d-66022aa7d036-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w\" (UID: \"c255c6e4-3338-4ded-bb5d-66022aa7d036\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w" Jul 08 19:14:49.539741 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:49.539724 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c255c6e4-3338-4ded-bb5d-66022aa7d036-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w\" (UID: \"c255c6e4-3338-4ded-bb5d-66022aa7d036\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w" Jul 08 19:14:49.541446 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:49.541426 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c255c6e4-3338-4ded-bb5d-66022aa7d036-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w\" (UID: \"c255c6e4-3338-4ded-bb5d-66022aa7d036\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w" Jul 08 19:14:49.541645 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:49.541628 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c255c6e4-3338-4ded-bb5d-66022aa7d036-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w\" (UID: \"c255c6e4-3338-4ded-bb5d-66022aa7d036\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w" Jul 08 19:14:49.550620 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:49.550600 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76jvc\" (UniqueName: \"kubernetes.io/projected/c255c6e4-3338-4ded-bb5d-66022aa7d036-kube-api-access-76jvc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w\" (UID: \"c255c6e4-3338-4ded-bb5d-66022aa7d036\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w" Jul 08 19:14:49.625688 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:49.625656 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w" Jul 08 19:14:49.771742 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:49.771719 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w"] Jul 08 19:14:49.773180 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:14:49.773155 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc255c6e4_3338_4ded_bb5d_66022aa7d036.slice/crio-6899e5f090fda01957b6902891d729a85cbb4460762f7b14abf95304584c30d2 WatchSource:0}: Error finding container 6899e5f090fda01957b6902891d729a85cbb4460762f7b14abf95304584c30d2: Status 404 returned error can't find the container with id 6899e5f090fda01957b6902891d729a85cbb4460762f7b14abf95304584c30d2 Jul 08 19:14:50.392620 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:50.392569 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w" event={"ID":"c255c6e4-3338-4ded-bb5d-66022aa7d036","Type":"ContainerStarted","Data":"f8f544ee9e75938719ebf00ba29e2bd0e1e2a8d5001c58a870080b2c4220bc7f"} Jul 08 19:14:50.393004 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:50.392687 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w" event={"ID":"c255c6e4-3338-4ded-bb5d-66022aa7d036","Type":"ContainerStarted","Data":"6899e5f090fda01957b6902891d729a85cbb4460762f7b14abf95304584c30d2"} Jul 08 19:14:50.415197 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:50.415150 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w" podStartSLOduration=1.244607373 podStartE2EDuration="1.415137592s" totalImagesPullingTime="170.530219ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:14:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:14:50.411985774 +0000 UTC m=+673.670938697" watchObservedRunningTime="2026-07-08 19:14:50.415137592 +0000 UTC m=+673.674090514" Jul 08 19:14:51.847883 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:51.847841 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:14:56.471839 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:56.471803 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:14:59.625939 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:59.625896 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w" Jul 08 19:14:59.626307 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:59.625969 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w" Jul 08 19:14:59.638739 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:14:59.638714 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w" Jul 08 19:15:00.158570 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:15:00.158533 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725635-xl7np"] Jul 08 19:15:00.161432 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:15:00.161415 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725635-xl7np" Jul 08 19:15:00.177354 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:15:00.177332 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725635-xl7np"] Jul 08 19:15:00.232072 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:15:00.232043 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdq7x\" (UniqueName: \"kubernetes.io/projected/df14e092-4e05-4d90-b72d-2e026a5f7f16-kube-api-access-zdq7x\") pod \"maas-api-key-cleanup-29725635-xl7np\" (UID: \"df14e092-4e05-4d90-b72d-2e026a5f7f16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725635-xl7np" Jul 08 19:15:00.332773 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:15:00.332742 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdq7x\" (UniqueName: \"kubernetes.io/projected/df14e092-4e05-4d90-b72d-2e026a5f7f16-kube-api-access-zdq7x\") pod \"maas-api-key-cleanup-29725635-xl7np\" (UID: \"df14e092-4e05-4d90-b72d-2e026a5f7f16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725635-xl7np" Jul 08 19:15:00.346114 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:15:00.346082 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdq7x\" (UniqueName: \"kubernetes.io/projected/df14e092-4e05-4d90-b72d-2e026a5f7f16-kube-api-access-zdq7x\") pod \"maas-api-key-cleanup-29725635-xl7np\" (UID: \"df14e092-4e05-4d90-b72d-2e026a5f7f16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725635-xl7np" Jul 08 19:15:00.459013 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:15:00.458931 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w" Jul 08 19:15:00.471982 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:15:00.471958 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725635-xl7np" Jul 08 19:15:00.606026 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:15:00.606003 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725635-xl7np"] Jul 08 19:15:00.608931 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:15:00.608906 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf14e092_4e05_4d90_b72d_2e026a5f7f16.slice/crio-59b3f468cd778eda548ecfa4486658b6bfd5e0a57cd741681135803a45ea1b6b WatchSource:0}: Error finding container 59b3f468cd778eda548ecfa4486658b6bfd5e0a57cd741681135803a45ea1b6b: Status 404 returned error can't find the container with id 59b3f468cd778eda548ecfa4486658b6bfd5e0a57cd741681135803a45ea1b6b Jul 08 19:15:01.455474 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:15:01.455430 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725635-xl7np" event={"ID":"df14e092-4e05-4d90-b72d-2e026a5f7f16","Type":"ContainerStarted","Data":"59b3f468cd778eda548ecfa4486658b6bfd5e0a57cd741681135803a45ea1b6b"} Jul 08 19:15:01.855725 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:15:01.855626 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:15:03.467410 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:15:03.467327 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725635-xl7np" event={"ID":"df14e092-4e05-4d90-b72d-2e026a5f7f16","Type":"ContainerStarted","Data":"2b1ffac64bc8b1c4d140a6ccbb9eb42a4f188c0b03b141333b1671444a09b65e"} Jul 08 19:15:03.487472 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:15:03.487409 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725635-xl7np" podStartSLOduration=1.210336169 podStartE2EDuration="3.48739132s" totalImagesPullingTime="2.277055151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:15:03.486233352 +0000 UTC m=+686.745186278" watchObservedRunningTime="2026-07-08 19:15:03.48739132 +0000 UTC m=+686.746344246" Jul 08 19:15:06.780061 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:15:06.780019 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:15:06.877513 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:15:06.877462 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:16:00.012685 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:16:00.012595 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725635-xl7np"] Jul 08 19:16:00.013200 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:16:00.012853 2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725635-xl7np" podUID="df14e092-4e05-4d90-b72d-2e026a5f7f16" containerName="cleanup" containerID="cri-o://2b1ffac64bc8b1c4d140a6ccbb9eb42a4f188c0b03b141333b1671444a09b65e" gracePeriod=30 Jul 08 19:16:09.965057 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:16:09.965018 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:16:10.068549 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:16:10.068507 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:16:11.277845 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:16:11.277807 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:16:11.366862 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:16:11.366826 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:16:27.513779 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:16:27.513737 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:16:30.211199 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:16:30.211164 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725635-xl7np" Jul 08 19:16:30.401084 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:16:30.401057 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zdq7x\" (UniqueName: \"kubernetes.io/projected/df14e092-4e05-4d90-b72d-2e026a5f7f16-kube-api-access-zdq7x\") pod \"df14e092-4e05-4d90-b72d-2e026a5f7f16\" (UID: \"df14e092-4e05-4d90-b72d-2e026a5f7f16\") " Jul 08 19:16:30.403149 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:16:30.403124 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df14e092-4e05-4d90-b72d-2e026a5f7f16-kube-api-access-zdq7x" (OuterVolumeSpecName: "kube-api-access-zdq7x") pod "df14e092-4e05-4d90-b72d-2e026a5f7f16" (UID: "df14e092-4e05-4d90-b72d-2e026a5f7f16"). InnerVolumeSpecName "kube-api-access-zdq7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:16:30.501931 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:16:30.501900 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdq7x\" (UniqueName: \"kubernetes.io/projected/df14e092-4e05-4d90-b72d-2e026a5f7f16-kube-api-access-zdq7x\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:16:30.907360 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:16:30.907331 2558 generic.go:356] "Generic (PLEG): container finished" podID="df14e092-4e05-4d90-b72d-2e026a5f7f16" containerID="2b1ffac64bc8b1c4d140a6ccbb9eb42a4f188c0b03b141333b1671444a09b65e" exitCode=137 Jul 08 19:16:30.907638 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:16:30.907377 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725635-xl7np" Jul 08 19:16:30.907638 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:16:30.907407 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725635-xl7np" event={"ID":"df14e092-4e05-4d90-b72d-2e026a5f7f16","Type":"ContainerDied","Data":"2b1ffac64bc8b1c4d140a6ccbb9eb42a4f188c0b03b141333b1671444a09b65e"} Jul 08 19:16:30.907638 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:16:30.907464 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725635-xl7np" event={"ID":"df14e092-4e05-4d90-b72d-2e026a5f7f16","Type":"ContainerDied","Data":"59b3f468cd778eda548ecfa4486658b6bfd5e0a57cd741681135803a45ea1b6b"} Jul 08 19:16:30.907638 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:16:30.907508 2558 scope.go:117] "RemoveContainer" containerID="2b1ffac64bc8b1c4d140a6ccbb9eb42a4f188c0b03b141333b1671444a09b65e" Jul 08 19:16:30.916313 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:16:30.916298 2558 scope.go:117] "RemoveContainer" containerID="2b1ffac64bc8b1c4d140a6ccbb9eb42a4f188c0b03b141333b1671444a09b65e" Jul 08 19:16:30.916568 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:16:30.916550 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b1ffac64bc8b1c4d140a6ccbb9eb42a4f188c0b03b141333b1671444a09b65e\": container with ID starting with 2b1ffac64bc8b1c4d140a6ccbb9eb42a4f188c0b03b141333b1671444a09b65e not found: ID does not exist" containerID="2b1ffac64bc8b1c4d140a6ccbb9eb42a4f188c0b03b141333b1671444a09b65e" Jul 08 19:16:30.916656 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:16:30.916588 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b1ffac64bc8b1c4d140a6ccbb9eb42a4f188c0b03b141333b1671444a09b65e"} err="failed to get container status \"2b1ffac64bc8b1c4d140a6ccbb9eb42a4f188c0b03b141333b1671444a09b65e\": rpc error: code = NotFound desc = could not find container \"2b1ffac64bc8b1c4d140a6ccbb9eb42a4f188c0b03b141333b1671444a09b65e\": container with ID starting with 2b1ffac64bc8b1c4d140a6ccbb9eb42a4f188c0b03b141333b1671444a09b65e not found: ID does not exist" Jul 08 19:16:30.929042 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:16:30.929023 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725635-xl7np"] Jul 08 19:16:30.932789 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:16:30.932769 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725635-xl7np"] Jul 08 19:16:31.282006 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:16:31.281920 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df14e092-4e05-4d90-b72d-2e026a5f7f16" path="/var/lib/kubelet/pods/df14e092-4e05-4d90-b72d-2e026a5f7f16/volumes" Jul 08 19:17:22.226517 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:22.226477 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:17:22.335237 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:22.335203 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-796d55d59-4t5zn"] Jul 08 19:17:22.335498 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:22.335450 2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-796d55d59-4t5zn" podUID="e978acd9-810b-4760-a61e-d3ad50573633" containerName="manager" containerID="cri-o://938a9b4c5393e6b5608c256a9f2d1ac608f2540d514ffcd951d25d30ddb26b92" gracePeriod=30 Jul 08 19:17:22.588307 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:22.588280 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-796d55d59-4t5zn" Jul 08 19:17:22.754994 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:22.754900 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-595bj\" (UniqueName: \"kubernetes.io/projected/e978acd9-810b-4760-a61e-d3ad50573633-kube-api-access-595bj\") pod \"e978acd9-810b-4760-a61e-d3ad50573633\" (UID: \"e978acd9-810b-4760-a61e-d3ad50573633\") " Jul 08 19:17:22.754994 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:22.754985 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e978acd9-810b-4760-a61e-d3ad50573633-webhook-cert\") pod \"e978acd9-810b-4760-a61e-d3ad50573633\" (UID: \"e978acd9-810b-4760-a61e-d3ad50573633\") " Jul 08 19:17:22.757070 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:22.757029 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e978acd9-810b-4760-a61e-d3ad50573633-kube-api-access-595bj" (OuterVolumeSpecName: "kube-api-access-595bj") pod "e978acd9-810b-4760-a61e-d3ad50573633" (UID: "e978acd9-810b-4760-a61e-d3ad50573633"). InnerVolumeSpecName "kube-api-access-595bj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:17:22.757181 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:22.757066 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e978acd9-810b-4760-a61e-d3ad50573633-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e978acd9-810b-4760-a61e-d3ad50573633" (UID: "e978acd9-810b-4760-a61e-d3ad50573633"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:17:22.855828 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:22.855793 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-595bj\" (UniqueName: \"kubernetes.io/projected/e978acd9-810b-4760-a61e-d3ad50573633-kube-api-access-595bj\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:17:22.855828 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:22.855819 2558 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e978acd9-810b-4760-a61e-d3ad50573633-webhook-cert\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:17:23.176032 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:23.175998 2558 generic.go:356] "Generic (PLEG): container finished" podID="e978acd9-810b-4760-a61e-d3ad50573633" containerID="938a9b4c5393e6b5608c256a9f2d1ac608f2540d514ffcd951d25d30ddb26b92" exitCode=0 Jul 08 19:17:23.176032 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:23.176031 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-796d55d59-4t5zn" Jul 08 19:17:23.176238 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:23.176115 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-796d55d59-4t5zn" event={"ID":"e978acd9-810b-4760-a61e-d3ad50573633","Type":"ContainerDied","Data":"938a9b4c5393e6b5608c256a9f2d1ac608f2540d514ffcd951d25d30ddb26b92"} Jul 08 19:17:23.176238 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:23.176161 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-796d55d59-4t5zn" event={"ID":"e978acd9-810b-4760-a61e-d3ad50573633","Type":"ContainerDied","Data":"ea16794a61d76cc69f296ba37af7e19c94c7ca09e2922799382d8a2716e5e9fb"} Jul 08 19:17:23.176238 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:23.176186 2558 scope.go:117] "RemoveContainer" containerID="938a9b4c5393e6b5608c256a9f2d1ac608f2540d514ffcd951d25d30ddb26b92" Jul 08 19:17:23.185628 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:23.185612 2558 scope.go:117] "RemoveContainer" containerID="938a9b4c5393e6b5608c256a9f2d1ac608f2540d514ffcd951d25d30ddb26b92" Jul 08 19:17:23.185871 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:17:23.185853 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"938a9b4c5393e6b5608c256a9f2d1ac608f2540d514ffcd951d25d30ddb26b92\": container with ID starting with 938a9b4c5393e6b5608c256a9f2d1ac608f2540d514ffcd951d25d30ddb26b92 not found: ID does not exist" containerID="938a9b4c5393e6b5608c256a9f2d1ac608f2540d514ffcd951d25d30ddb26b92" Jul 08 19:17:23.185930 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:23.185881 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"938a9b4c5393e6b5608c256a9f2d1ac608f2540d514ffcd951d25d30ddb26b92"} err="failed to get container status \"938a9b4c5393e6b5608c256a9f2d1ac608f2540d514ffcd951d25d30ddb26b92\": rpc error: code = NotFound desc = could not find container \"938a9b4c5393e6b5608c256a9f2d1ac608f2540d514ffcd951d25d30ddb26b92\": container with ID starting with 938a9b4c5393e6b5608c256a9f2d1ac608f2540d514ffcd951d25d30ddb26b92 not found: ID does not exist" Jul 08 19:17:23.200922 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:23.200900 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-796d55d59-4t5zn"] Jul 08 19:17:23.206716 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:23.206695 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-796d55d59-4t5zn"] Jul 08 19:17:23.281763 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:23.281738 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e978acd9-810b-4760-a61e-d3ad50573633" path="/var/lib/kubelet/pods/e978acd9-810b-4760-a61e-d3ad50573633/volumes" Jul 08 19:17:23.913260 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:23.913217 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:17:24.040035 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:24.039997 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-796d55d59-7p887"] Jul 08 19:17:24.041090 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:24.041062 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e978acd9-810b-4760-a61e-d3ad50573633" containerName="manager" Jul 08 19:17:24.041250 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:24.041239 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e978acd9-810b-4760-a61e-d3ad50573633" containerName="manager" Jul 08 19:17:24.041379 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:24.041369 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df14e092-4e05-4d90-b72d-2e026a5f7f16" containerName="cleanup" Jul 08 19:17:24.041449 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:24.041441 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="df14e092-4e05-4d90-b72d-2e026a5f7f16" containerName="cleanup" Jul 08 19:17:24.041695 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:24.041679 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e978acd9-810b-4760-a61e-d3ad50573633" containerName="manager" Jul 08 19:17:24.041821 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:24.041811 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="df14e092-4e05-4d90-b72d-2e026a5f7f16" containerName="cleanup" Jul 08 19:17:24.047046 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:24.047005 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-796d55d59-7p887" Jul 08 19:17:24.049861 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:24.049837 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-796d55d59-7p887"] Jul 08 19:17:24.050304 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:24.050284 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 08 19:17:24.051312 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:24.051287 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-92bhq\"" Jul 08 19:17:24.051441 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:24.051340 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 08 19:17:24.168370 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:24.168294 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07700279-0333-4aee-ab4b-2ac005a6ec83-webhook-cert\") pod \"maas-controller-796d55d59-7p887\" (UID: \"07700279-0333-4aee-ab4b-2ac005a6ec83\") " pod="opendatahub/maas-controller-796d55d59-7p887" Jul 08 19:17:24.168370 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:24.168349 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5sbd\" (UniqueName: \"kubernetes.io/projected/07700279-0333-4aee-ab4b-2ac005a6ec83-kube-api-access-w5sbd\") pod \"maas-controller-796d55d59-7p887\" (UID: \"07700279-0333-4aee-ab4b-2ac005a6ec83\") " pod="opendatahub/maas-controller-796d55d59-7p887" Jul 08 19:17:24.269569 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:24.269532 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07700279-0333-4aee-ab4b-2ac005a6ec83-webhook-cert\") pod \"maas-controller-796d55d59-7p887\" (UID: \"07700279-0333-4aee-ab4b-2ac005a6ec83\") " pod="opendatahub/maas-controller-796d55d59-7p887" Jul 08 19:17:24.269778 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:24.269627 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5sbd\" (UniqueName: \"kubernetes.io/projected/07700279-0333-4aee-ab4b-2ac005a6ec83-kube-api-access-w5sbd\") pod \"maas-controller-796d55d59-7p887\" (UID: \"07700279-0333-4aee-ab4b-2ac005a6ec83\") " pod="opendatahub/maas-controller-796d55d59-7p887" Jul 08 19:17:24.272002 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:24.271983 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07700279-0333-4aee-ab4b-2ac005a6ec83-webhook-cert\") pod \"maas-controller-796d55d59-7p887\" (UID: \"07700279-0333-4aee-ab4b-2ac005a6ec83\") " pod="opendatahub/maas-controller-796d55d59-7p887" Jul 08 19:17:24.278201 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:24.278176 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5sbd\" (UniqueName: \"kubernetes.io/projected/07700279-0333-4aee-ab4b-2ac005a6ec83-kube-api-access-w5sbd\") pod \"maas-controller-796d55d59-7p887\" (UID: \"07700279-0333-4aee-ab4b-2ac005a6ec83\") " pod="opendatahub/maas-controller-796d55d59-7p887" Jul 08 19:17:24.359853 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:24.359823 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-796d55d59-7p887" Jul 08 19:17:24.490860 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:24.490835 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-796d55d59-7p887"] Jul 08 19:17:24.493427 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:17:24.493397 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07700279_0333_4aee_ab4b_2ac005a6ec83.slice/crio-e8b89ba3355c7514b7ee596b70e210135827906ac673631aeccb10d0e0d66cd5 WatchSource:0}: Error finding container e8b89ba3355c7514b7ee596b70e210135827906ac673631aeccb10d0e0d66cd5: Status 404 returned error can't find the container with id e8b89ba3355c7514b7ee596b70e210135827906ac673631aeccb10d0e0d66cd5 Jul 08 19:17:25.191530 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:25.191497 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-796d55d59-7p887" event={"ID":"07700279-0333-4aee-ab4b-2ac005a6ec83","Type":"ContainerStarted","Data":"e8b89ba3355c7514b7ee596b70e210135827906ac673631aeccb10d0e0d66cd5"} Jul 08 19:17:25.330979 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:25.330932 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:17:26.198235 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:26.198202 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-796d55d59-7p887" event={"ID":"07700279-0333-4aee-ab4b-2ac005a6ec83","Type":"ContainerStarted","Data":"09c5befd258b585a009ad5da64cbee61f04c9d064462eaa60346e38205184890"} Jul 08 19:17:26.198681 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:26.198390 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-796d55d59-7p887" Jul 08 19:17:26.217639 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:26.217595 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-796d55d59-7p887" podStartSLOduration=1.511147683 podStartE2EDuration="2.217569077s" totalImagesPullingTime="706.421394ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:17:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:17:26.215392489 +0000 UTC m=+829.474345415" watchObservedRunningTime="2026-07-08 19:17:26.217569077 +0000 UTC m=+829.476522012" Jul 08 19:17:26.364544 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:26.364509 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:17:37.208533 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:17:37.208500 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-796d55d59-7p887" Jul 08 19:18:05.916626 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:18:05.916562 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:18:06.018493 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:18:06.018458 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:18:06.152594 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:18:06.152554 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:18:12.208715 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:18:12.208677 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:18:12.314441 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:18:12.314408 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:18:12.421403 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:18:12.421366 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:18:12.604768 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:18:12.604681 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:18:13.641387 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:18:13.641344 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:18:13.745638 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:18:13.745601 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:18:18.609829 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:18:18.609755 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:18:18.692208 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:18:18.692153 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:18:37.219691 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:18:37.219617 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-54hnr_1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a/console-operator/2.log" Jul 08 19:18:37.220254 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:18:37.219807 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-54hnr_1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a/console-operator/2.log" Jul 08 19:21:00.924965 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:21:00.924931 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:22:54.736039 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:22:54.736004 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:23:05.075943 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:23:05.075824 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:23:05.131942 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:23:05.131905 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:23:37.252289 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:23:37.252258 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-54hnr_1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a/console-operator/2.log" Jul 08 19:23:37.252896 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:23:37.252313 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-54hnr_1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a/console-operator/2.log" Jul 08 19:23:47.295761 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:23:47.295727 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:23:47.396050 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:23:47.396019 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:23:49.257270 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:23:49.257238 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:23:49.358965 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:23:49.358931 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:24:34.936794 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:24:34.936710 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:28:37.294353 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:28:37.294326 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-54hnr_1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a/console-operator/2.log" Jul 08 19:28:37.294917 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:28:37.294566 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-54hnr_1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a/console-operator/2.log" Jul 08 19:29:17.760314 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:17.760230 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:26.368187 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:26.368147 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:34.620171 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:34.620113 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:39.716741 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:39.716705 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:42.504233 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:42.504196 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:42.605048 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:42.605012 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:42.956746 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:42.956712 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:48.766151 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:48.766109 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:48.834846 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:48.834807 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:49.053086 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:49.053013 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:49.147914 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:49.147880 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:50.161401 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:50.161362 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:50.240978 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:50.240942 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:50.242406 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:50.242383 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-39a4045b-55ddfdcfdf-cjcvv"] Jul 08 19:29:50.245827 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:50.245809 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-39a4045b-55ddfdcfdf-cjcvv" Jul 08 19:29:50.247980 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:50.247959 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-39a4045b-serving-cert\"" Jul 08 19:29:50.256785 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:50.256765 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-39a4045b-55ddfdcfdf-cjcvv"] Jul 08 19:29:50.306924 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:50.306886 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:50.353202 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:50.353165 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae6bb36d-254e-40e8-9b32-6c060ef70e8a-openshift-service-ca\") pod \"maas-api-e2e-ait-39a4045b-55ddfdcfdf-cjcvv\" (UID: \"ae6bb36d-254e-40e8-9b32-6c060ef70e8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-39a4045b-55ddfdcfdf-cjcvv" Jul 08 19:29:50.353456 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:50.353425 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae6bb36d-254e-40e8-9b32-6c060ef70e8a-maas-api-tls\") pod \"maas-api-e2e-ait-39a4045b-55ddfdcfdf-cjcvv\" (UID: \"ae6bb36d-254e-40e8-9b32-6c060ef70e8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-39a4045b-55ddfdcfdf-cjcvv" Jul 08 19:29:50.353553 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:50.353484 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbvc5\" (UniqueName: \"kubernetes.io/projected/ae6bb36d-254e-40e8-9b32-6c060ef70e8a-kube-api-access-tbvc5\") pod \"maas-api-e2e-ait-39a4045b-55ddfdcfdf-cjcvv\" (UID: \"ae6bb36d-254e-40e8-9b32-6c060ef70e8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-39a4045b-55ddfdcfdf-cjcvv" Jul 08 19:29:50.454538 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:50.454438 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae6bb36d-254e-40e8-9b32-6c060ef70e8a-openshift-service-ca\") pod \"maas-api-e2e-ait-39a4045b-55ddfdcfdf-cjcvv\" (UID: \"ae6bb36d-254e-40e8-9b32-6c060ef70e8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-39a4045b-55ddfdcfdf-cjcvv" Jul 08 19:29:50.454538 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:50.454488 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae6bb36d-254e-40e8-9b32-6c060ef70e8a-maas-api-tls\") pod \"maas-api-e2e-ait-39a4045b-55ddfdcfdf-cjcvv\" (UID: \"ae6bb36d-254e-40e8-9b32-6c060ef70e8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-39a4045b-55ddfdcfdf-cjcvv" Jul 08 19:29:50.454538 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:50.454526 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbvc5\" (UniqueName: \"kubernetes.io/projected/ae6bb36d-254e-40e8-9b32-6c060ef70e8a-kube-api-access-tbvc5\") pod \"maas-api-e2e-ait-39a4045b-55ddfdcfdf-cjcvv\" (UID: \"ae6bb36d-254e-40e8-9b32-6c060ef70e8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-39a4045b-55ddfdcfdf-cjcvv" Jul 08 19:29:50.454848 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:29:50.454698 2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-39a4045b-serving-cert: secret "maas-api-e2e-ait-39a4045b-serving-cert" not found Jul 08 19:29:50.454848 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:29:50.454773 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae6bb36d-254e-40e8-9b32-6c060ef70e8a-maas-api-tls podName:ae6bb36d-254e-40e8-9b32-6c060ef70e8a nodeName:}" failed. No retries permitted until 2026-07-08 19:29:50.95475452 +0000 UTC m=+1574.213707424 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ae6bb36d-254e-40e8-9b32-6c060ef70e8a-maas-api-tls") pod "maas-api-e2e-ait-39a4045b-55ddfdcfdf-cjcvv" (UID: "ae6bb36d-254e-40e8-9b32-6c060ef70e8a") : secret "maas-api-e2e-ait-39a4045b-serving-cert" not found Jul 08 19:29:50.455224 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:50.455198 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae6bb36d-254e-40e8-9b32-6c060ef70e8a-openshift-service-ca\") pod \"maas-api-e2e-ait-39a4045b-55ddfdcfdf-cjcvv\" (UID: \"ae6bb36d-254e-40e8-9b32-6c060ef70e8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-39a4045b-55ddfdcfdf-cjcvv" Jul 08 19:29:50.466194 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:50.466163 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbvc5\" (UniqueName: \"kubernetes.io/projected/ae6bb36d-254e-40e8-9b32-6c060ef70e8a-kube-api-access-tbvc5\") pod \"maas-api-e2e-ait-39a4045b-55ddfdcfdf-cjcvv\" (UID: \"ae6bb36d-254e-40e8-9b32-6c060ef70e8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-39a4045b-55ddfdcfdf-cjcvv" Jul 08 19:29:50.756105 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:50.756019 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:50.856015 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:50.855970 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:50.956701 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:50.956670 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:50.958761 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:50.958736 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae6bb36d-254e-40e8-9b32-6c060ef70e8a-maas-api-tls\") pod \"maas-api-e2e-ait-39a4045b-55ddfdcfdf-cjcvv\" (UID: \"ae6bb36d-254e-40e8-9b32-6c060ef70e8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-39a4045b-55ddfdcfdf-cjcvv" Jul 08 19:29:50.961107 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:50.961084 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae6bb36d-254e-40e8-9b32-6c060ef70e8a-maas-api-tls\") pod \"maas-api-e2e-ait-39a4045b-55ddfdcfdf-cjcvv\" (UID: \"ae6bb36d-254e-40e8-9b32-6c060ef70e8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-39a4045b-55ddfdcfdf-cjcvv" Jul 08 19:29:51.157010 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:51.156966 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-39a4045b-55ddfdcfdf-cjcvv" Jul 08 19:29:51.290733 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:51.290705 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-39a4045b-55ddfdcfdf-cjcvv"] Jul 08 19:29:51.291816 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:29:51.291791 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae6bb36d_254e_40e8_9b32_6c060ef70e8a.slice/crio-620615d1fcfcb94a97bc395d949d02accac6951ca7dc6380bb97079667ffeabe WatchSource:0}: Error finding container 620615d1fcfcb94a97bc395d949d02accac6951ca7dc6380bb97079667ffeabe: Status 404 returned error can't find the container with id 620615d1fcfcb94a97bc395d949d02accac6951ca7dc6380bb97079667ffeabe Jul 08 19:29:51.293076 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:51.293054 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 19:29:51.982903 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:51.982869 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-39a4045b-55ddfdcfdf-cjcvv" event={"ID":"ae6bb36d-254e-40e8-9b32-6c060ef70e8a","Type":"ContainerStarted","Data":"620615d1fcfcb94a97bc395d949d02accac6951ca7dc6380bb97079667ffeabe"} Jul 08 19:29:52.361472 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:52.361433 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:52.995626 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:52.995565 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-39a4045b-55ddfdcfdf-cjcvv" event={"ID":"ae6bb36d-254e-40e8-9b32-6c060ef70e8a","Type":"ContainerStarted","Data":"c573f4428fcb7c2f9380822733138be039fb01f19e4bc824607aa7e43bf46f77"} Jul 08 19:29:52.995821 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:52.995768 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-39a4045b-55ddfdcfdf-cjcvv" Jul 08 19:29:53.016487 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:53.016431 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-39a4045b-55ddfdcfdf-cjcvv" podStartSLOduration=2.14460658 podStartE2EDuration="3.016416213s" totalImagesPullingTime="871.809633ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:29:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:29:53.015744601 +0000 UTC m=+1576.274697528" watchObservedRunningTime="2026-07-08 19:29:53.016416213 +0000 UTC m=+1576.275369138" Jul 08 19:29:55.199547 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:55.199488 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:55.417921 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:55.417887 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:55.516463 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:55.516374 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:55.612645 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:55.612607 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-39a4045b-55ddfdcfdf-cjcvv"] Jul 08 19:29:55.612897 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:55.612870 2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-39a4045b-55ddfdcfdf-cjcvv" podUID="ae6bb36d-254e-40e8-9b32-6c060ef70e8a" containerName="maas-api" containerID="cri-o://c573f4428fcb7c2f9380822733138be039fb01f19e4bc824607aa7e43bf46f77" gracePeriod=30 Jul 08 19:29:55.636883 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:55.636850 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:29:55.859073 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:55.859049 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-39a4045b-55ddfdcfdf-cjcvv" Jul 08 19:29:56.003743 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:56.003705 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae6bb36d-254e-40e8-9b32-6c060ef70e8a-openshift-service-ca\") pod \"ae6bb36d-254e-40e8-9b32-6c060ef70e8a\" (UID: \"ae6bb36d-254e-40e8-9b32-6c060ef70e8a\") " Jul 08 19:29:56.003925 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:56.003800 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae6bb36d-254e-40e8-9b32-6c060ef70e8a-maas-api-tls\") pod \"ae6bb36d-254e-40e8-9b32-6c060ef70e8a\" (UID: \"ae6bb36d-254e-40e8-9b32-6c060ef70e8a\") " Jul 08 19:29:56.003925 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:56.003864 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbvc5\" (UniqueName: \"kubernetes.io/projected/ae6bb36d-254e-40e8-9b32-6c060ef70e8a-kube-api-access-tbvc5\") pod \"ae6bb36d-254e-40e8-9b32-6c060ef70e8a\" (UID: \"ae6bb36d-254e-40e8-9b32-6c060ef70e8a\") " Jul 08 19:29:56.004131 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:56.004100 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae6bb36d-254e-40e8-9b32-6c060ef70e8a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ae6bb36d-254e-40e8-9b32-6c060ef70e8a" (UID: "ae6bb36d-254e-40e8-9b32-6c060ef70e8a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:29:56.005840 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:56.005816 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae6bb36d-254e-40e8-9b32-6c060ef70e8a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ae6bb36d-254e-40e8-9b32-6c060ef70e8a" (UID: "ae6bb36d-254e-40e8-9b32-6c060ef70e8a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:29:56.005950 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:56.005919 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae6bb36d-254e-40e8-9b32-6c060ef70e8a-kube-api-access-tbvc5" (OuterVolumeSpecName: "kube-api-access-tbvc5") pod "ae6bb36d-254e-40e8-9b32-6c060ef70e8a" (UID: "ae6bb36d-254e-40e8-9b32-6c060ef70e8a"). InnerVolumeSpecName "kube-api-access-tbvc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:29:56.011348 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:56.011325 2558 generic.go:356] "Generic (PLEG): container finished" podID="ae6bb36d-254e-40e8-9b32-6c060ef70e8a" containerID="c573f4428fcb7c2f9380822733138be039fb01f19e4bc824607aa7e43bf46f77" exitCode=0 Jul 08 19:29:56.011446 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:56.011386 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-39a4045b-55ddfdcfdf-cjcvv" Jul 08 19:29:56.011446 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:56.011404 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-39a4045b-55ddfdcfdf-cjcvv" event={"ID":"ae6bb36d-254e-40e8-9b32-6c060ef70e8a","Type":"ContainerDied","Data":"c573f4428fcb7c2f9380822733138be039fb01f19e4bc824607aa7e43bf46f77"} Jul 08 19:29:56.011557 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:56.011454 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-39a4045b-55ddfdcfdf-cjcvv" event={"ID":"ae6bb36d-254e-40e8-9b32-6c060ef70e8a","Type":"ContainerDied","Data":"620615d1fcfcb94a97bc395d949d02accac6951ca7dc6380bb97079667ffeabe"} Jul 08 19:29:56.011557 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:56.011485 2558 scope.go:117] "RemoveContainer" containerID="c573f4428fcb7c2f9380822733138be039fb01f19e4bc824607aa7e43bf46f77" Jul 08 19:29:56.029701 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:56.029678 2558 scope.go:117] "RemoveContainer" containerID="c573f4428fcb7c2f9380822733138be039fb01f19e4bc824607aa7e43bf46f77" Jul 08 19:29:56.030024 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:29:56.030004 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c573f4428fcb7c2f9380822733138be039fb01f19e4bc824607aa7e43bf46f77\": container with ID starting with c573f4428fcb7c2f9380822733138be039fb01f19e4bc824607aa7e43bf46f77 not found: ID does not exist" containerID="c573f4428fcb7c2f9380822733138be039fb01f19e4bc824607aa7e43bf46f77" Jul 08 19:29:56.030097 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:56.030035 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c573f4428fcb7c2f9380822733138be039fb01f19e4bc824607aa7e43bf46f77"} err="failed to get container status \"c573f4428fcb7c2f9380822733138be039fb01f19e4bc824607aa7e43bf46f77\": rpc error: code = NotFound desc = could not find container \"c573f4428fcb7c2f9380822733138be039fb01f19e4bc824607aa7e43bf46f77\": container with ID starting with c573f4428fcb7c2f9380822733138be039fb01f19e4bc824607aa7e43bf46f77 not found: ID does not exist" Jul 08 19:29:56.040135 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:56.040082 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-39a4045b-55ddfdcfdf-cjcvv"] Jul 08 19:29:56.043979 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:56.043959 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-39a4045b-55ddfdcfdf-cjcvv"] Jul 08 19:29:56.105303 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:56.105275 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae6bb36d-254e-40e8-9b32-6c060ef70e8a-maas-api-tls\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:29:56.105303 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:56.105301 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbvc5\" (UniqueName: \"kubernetes.io/projected/ae6bb36d-254e-40e8-9b32-6c060ef70e8a-kube-api-access-tbvc5\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:29:56.105480 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:56.105310 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae6bb36d-254e-40e8-9b32-6c060ef70e8a-openshift-service-ca\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:29:57.281645 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:57.281605 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae6bb36d-254e-40e8-9b32-6c060ef70e8a" path="/var/lib/kubelet/pods/ae6bb36d-254e-40e8-9b32-6c060ef70e8a/volumes" Jul 08 19:29:58.583771 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:29:58.583735 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:00.133971 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:00.133937 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725650-tctpl"] Jul 08 19:30:00.134349 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:00.134304 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae6bb36d-254e-40e8-9b32-6c060ef70e8a" containerName="maas-api" Jul 08 19:30:00.134349 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:00.134316 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae6bb36d-254e-40e8-9b32-6c060ef70e8a" containerName="maas-api" Jul 08 19:30:00.134430 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:00.134419 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae6bb36d-254e-40e8-9b32-6c060ef70e8a" containerName="maas-api" Jul 08 19:30:00.139207 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:00.139033 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725650-tctpl" Jul 08 19:30:00.149403 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:00.149184 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725650-tctpl"] Jul 08 19:30:00.240505 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:00.240463 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cm9t\" (UniqueName: \"kubernetes.io/projected/985a70ac-d361-4e9e-afbb-af65bd49d584-kube-api-access-2cm9t\") pod \"maas-api-key-cleanup-29725650-tctpl\" (UID: \"985a70ac-d361-4e9e-afbb-af65bd49d584\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725650-tctpl" Jul 08 19:30:00.341410 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:00.341379 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cm9t\" (UniqueName: \"kubernetes.io/projected/985a70ac-d361-4e9e-afbb-af65bd49d584-kube-api-access-2cm9t\") pod \"maas-api-key-cleanup-29725650-tctpl\" (UID: \"985a70ac-d361-4e9e-afbb-af65bd49d584\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725650-tctpl" Jul 08 19:30:00.350362 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:00.350326 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cm9t\" (UniqueName: \"kubernetes.io/projected/985a70ac-d361-4e9e-afbb-af65bd49d584-kube-api-access-2cm9t\") pod \"maas-api-key-cleanup-29725650-tctpl\" (UID: \"985a70ac-d361-4e9e-afbb-af65bd49d584\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725650-tctpl" Jul 08 19:30:00.453316 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:00.453230 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725650-tctpl" Jul 08 19:30:00.588939 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:00.587499 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725650-tctpl"] Jul 08 19:30:00.592946 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:30:00.592921 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod985a70ac_d361_4e9e_afbb_af65bd49d584.slice/crio-8cc467f506ad4350dca6c0cbfc4a28a076c7504a9872b28b0cdd1bbd739faa7d WatchSource:0}: Error finding container 8cc467f506ad4350dca6c0cbfc4a28a076c7504a9872b28b0cdd1bbd739faa7d: Status 404 returned error can't find the container with id 8cc467f506ad4350dca6c0cbfc4a28a076c7504a9872b28b0cdd1bbd739faa7d Jul 08 19:30:01.040552 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:01.040512 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725650-tctpl" event={"ID":"985a70ac-d361-4e9e-afbb-af65bd49d584","Type":"ContainerStarted","Data":"a44de3b47b439555201507f9ab1f139a30d42a41b40b8d4a09caea1c924a6a2e"} Jul 08 19:30:01.040760 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:01.040597 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725650-tctpl" event={"ID":"985a70ac-d361-4e9e-afbb-af65bd49d584","Type":"ContainerStarted","Data":"8cc467f506ad4350dca6c0cbfc4a28a076c7504a9872b28b0cdd1bbd739faa7d"} Jul 08 19:30:01.057886 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:01.057831 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725650-tctpl" podStartSLOduration=1.057815562 podStartE2EDuration="1.057815562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:30:01.055426186 +0000 UTC m=+1584.314379134" watchObservedRunningTime="2026-07-08 19:30:01.057815562 +0000 UTC m=+1584.316768486" Jul 08 19:30:01.615382 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:01.615347 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:01.718473 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:01.718430 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:01.831802 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:01.831765 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:01.929389 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:01.929349 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:03.014035 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:03.014000 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:03.089887 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:03.089852 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:03.091348 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:03.091316 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-19b7c553-8655fc775b-tsqgr"] Jul 08 19:30:03.095545 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:03.095517 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19b7c553-8655fc775b-tsqgr" Jul 08 19:30:03.098369 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:03.098352 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-19b7c553-serving-cert\"" Jul 08 19:30:03.105867 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:03.105843 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-19b7c553-8655fc775b-tsqgr"] Jul 08 19:30:03.160874 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:03.160827 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:03.261691 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:03.261649 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:03.265834 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:03.265768 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3271ae7b-4ca6-4975-836e-88906ac8a837-openshift-service-ca\") pod \"maas-api-e2e-ait-19b7c553-8655fc775b-tsqgr\" (UID: \"3271ae7b-4ca6-4975-836e-88906ac8a837\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19b7c553-8655fc775b-tsqgr" Jul 08 19:30:03.265939 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:03.265841 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x829t\" (UniqueName: \"kubernetes.io/projected/3271ae7b-4ca6-4975-836e-88906ac8a837-kube-api-access-x829t\") pod \"maas-api-e2e-ait-19b7c553-8655fc775b-tsqgr\" (UID: \"3271ae7b-4ca6-4975-836e-88906ac8a837\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19b7c553-8655fc775b-tsqgr" Jul 08 19:30:03.265939 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:03.265877 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3271ae7b-4ca6-4975-836e-88906ac8a837-maas-api-tls\") pod \"maas-api-e2e-ait-19b7c553-8655fc775b-tsqgr\" (UID: \"3271ae7b-4ca6-4975-836e-88906ac8a837\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19b7c553-8655fc775b-tsqgr" Jul 08 19:30:03.367094 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:03.367057 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x829t\" (UniqueName: \"kubernetes.io/projected/3271ae7b-4ca6-4975-836e-88906ac8a837-kube-api-access-x829t\") pod \"maas-api-e2e-ait-19b7c553-8655fc775b-tsqgr\" (UID: \"3271ae7b-4ca6-4975-836e-88906ac8a837\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19b7c553-8655fc775b-tsqgr" Jul 08 19:30:03.367298 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:03.367106 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3271ae7b-4ca6-4975-836e-88906ac8a837-maas-api-tls\") pod \"maas-api-e2e-ait-19b7c553-8655fc775b-tsqgr\" (UID: \"3271ae7b-4ca6-4975-836e-88906ac8a837\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19b7c553-8655fc775b-tsqgr" Jul 08 19:30:03.367298 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:03.367151 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3271ae7b-4ca6-4975-836e-88906ac8a837-openshift-service-ca\") pod \"maas-api-e2e-ait-19b7c553-8655fc775b-tsqgr\" (UID: \"3271ae7b-4ca6-4975-836e-88906ac8a837\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19b7c553-8655fc775b-tsqgr" Jul 08 19:30:03.367865 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:03.367843 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3271ae7b-4ca6-4975-836e-88906ac8a837-openshift-service-ca\") pod \"maas-api-e2e-ait-19b7c553-8655fc775b-tsqgr\" (UID: \"3271ae7b-4ca6-4975-836e-88906ac8a837\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19b7c553-8655fc775b-tsqgr" Jul 08 19:30:03.369528 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:03.369509 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3271ae7b-4ca6-4975-836e-88906ac8a837-maas-api-tls\") pod \"maas-api-e2e-ait-19b7c553-8655fc775b-tsqgr\" (UID: \"3271ae7b-4ca6-4975-836e-88906ac8a837\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19b7c553-8655fc775b-tsqgr" Jul 08 19:30:03.377019 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:03.376999 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x829t\" (UniqueName: \"kubernetes.io/projected/3271ae7b-4ca6-4975-836e-88906ac8a837-kube-api-access-x829t\") pod \"maas-api-e2e-ait-19b7c553-8655fc775b-tsqgr\" (UID: \"3271ae7b-4ca6-4975-836e-88906ac8a837\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19b7c553-8655fc775b-tsqgr" Jul 08 19:30:03.407815 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:03.407788 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19b7c553-8655fc775b-tsqgr" Jul 08 19:30:03.545287 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:03.545202 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-19b7c553-8655fc775b-tsqgr"] Jul 08 19:30:03.547703 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:30:03.547677 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3271ae7b_4ca6_4975_836e_88906ac8a837.slice/crio-d30282a9ba76a51a089d8c4b8cae17c56fdfccb7cb654986f5d73bc11f3452b5 WatchSource:0}: Error finding container d30282a9ba76a51a089d8c4b8cae17c56fdfccb7cb654986f5d73bc11f3452b5: Status 404 returned error can't find the container with id d30282a9ba76a51a089d8c4b8cae17c56fdfccb7cb654986f5d73bc11f3452b5 Jul 08 19:30:03.568315 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:03.568286 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:03.670004 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:03.669962 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:04.058192 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:04.058150 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19b7c553-8655fc775b-tsqgr" event={"ID":"3271ae7b-4ca6-4975-836e-88906ac8a837","Type":"ContainerStarted","Data":"d30282a9ba76a51a089d8c4b8cae17c56fdfccb7cb654986f5d73bc11f3452b5"} Jul 08 19:30:05.065156 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:05.065114 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19b7c553-8655fc775b-tsqgr" event={"ID":"3271ae7b-4ca6-4975-836e-88906ac8a837","Type":"ContainerStarted","Data":"43484ea783e6f0738cb4850613b2f8ebeea7e49582df05ce7f1c01cf53bf0487"} Jul 08 19:30:05.065671 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:05.065251 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19b7c553-8655fc775b-tsqgr" Jul 08 19:30:05.083192 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:05.083143 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19b7c553-8655fc775b-tsqgr" podStartSLOduration=1.235844144 podStartE2EDuration="2.083127536s" totalImagesPullingTime="847.283392ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:30:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:30:05.081700015 +0000 UTC m=+1588.340652944" watchObservedRunningTime="2026-07-08 19:30:05.083127536 +0000 UTC m=+1588.342080460" Jul 08 19:30:05.366407 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:05.366354 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:08.061256 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:08.061218 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:08.365610 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:08.365515 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-19b7c553-8655fc775b-tsqgr"] Jul 08 19:30:08.365764 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:08.365713 2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19b7c553-8655fc775b-tsqgr" podUID="3271ae7b-4ca6-4975-836e-88906ac8a837" containerName="maas-api" containerID="cri-o://43484ea783e6f0738cb4850613b2f8ebeea7e49582df05ce7f1c01cf53bf0487" gracePeriod=30 Jul 08 19:30:08.393463 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:08.393429 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:08.614298 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:08.614270 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19b7c553-8655fc775b-tsqgr" Jul 08 19:30:08.715165 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:08.715129 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3271ae7b-4ca6-4975-836e-88906ac8a837-openshift-service-ca\") pod \"3271ae7b-4ca6-4975-836e-88906ac8a837\" (UID: \"3271ae7b-4ca6-4975-836e-88906ac8a837\") " Jul 08 19:30:08.715368 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:08.715233 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3271ae7b-4ca6-4975-836e-88906ac8a837-maas-api-tls\") pod \"3271ae7b-4ca6-4975-836e-88906ac8a837\" (UID: \"3271ae7b-4ca6-4975-836e-88906ac8a837\") " Jul 08 19:30:08.715368 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:08.715281 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x829t\" (UniqueName: \"kubernetes.io/projected/3271ae7b-4ca6-4975-836e-88906ac8a837-kube-api-access-x829t\") pod \"3271ae7b-4ca6-4975-836e-88906ac8a837\" (UID: \"3271ae7b-4ca6-4975-836e-88906ac8a837\") " Jul 08 19:30:08.715634 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:08.715564 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3271ae7b-4ca6-4975-836e-88906ac8a837-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3271ae7b-4ca6-4975-836e-88906ac8a837" (UID: "3271ae7b-4ca6-4975-836e-88906ac8a837"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:30:08.717289 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:08.717260 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3271ae7b-4ca6-4975-836e-88906ac8a837-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3271ae7b-4ca6-4975-836e-88906ac8a837" (UID: "3271ae7b-4ca6-4975-836e-88906ac8a837"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:30:08.717378 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:08.717335 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3271ae7b-4ca6-4975-836e-88906ac8a837-kube-api-access-x829t" (OuterVolumeSpecName: "kube-api-access-x829t") pod "3271ae7b-4ca6-4975-836e-88906ac8a837" (UID: "3271ae7b-4ca6-4975-836e-88906ac8a837"). InnerVolumeSpecName "kube-api-access-x829t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:30:08.816916 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:08.816880 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3271ae7b-4ca6-4975-836e-88906ac8a837-maas-api-tls\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:30:08.816916 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:08.816910 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x829t\" (UniqueName: \"kubernetes.io/projected/3271ae7b-4ca6-4975-836e-88906ac8a837-kube-api-access-x829t\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:30:08.816916 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:08.816922 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3271ae7b-4ca6-4975-836e-88906ac8a837-openshift-service-ca\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:30:09.086669 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:09.086554 2558 generic.go:356] "Generic (PLEG): container finished" podID="3271ae7b-4ca6-4975-836e-88906ac8a837" containerID="43484ea783e6f0738cb4850613b2f8ebeea7e49582df05ce7f1c01cf53bf0487" exitCode=0 Jul 08 19:30:09.086669 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:09.086620 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19b7c553-8655fc775b-tsqgr" Jul 08 19:30:09.086669 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:09.086647 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19b7c553-8655fc775b-tsqgr" event={"ID":"3271ae7b-4ca6-4975-836e-88906ac8a837","Type":"ContainerDied","Data":"43484ea783e6f0738cb4850613b2f8ebeea7e49582df05ce7f1c01cf53bf0487"} Jul 08 19:30:09.087171 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:09.086700 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19b7c553-8655fc775b-tsqgr" event={"ID":"3271ae7b-4ca6-4975-836e-88906ac8a837","Type":"ContainerDied","Data":"d30282a9ba76a51a089d8c4b8cae17c56fdfccb7cb654986f5d73bc11f3452b5"} Jul 08 19:30:09.087171 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:09.086723 2558 scope.go:117] "RemoveContainer" containerID="43484ea783e6f0738cb4850613b2f8ebeea7e49582df05ce7f1c01cf53bf0487" Jul 08 19:30:09.096463 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:09.096440 2558 scope.go:117] "RemoveContainer" containerID="43484ea783e6f0738cb4850613b2f8ebeea7e49582df05ce7f1c01cf53bf0487" Jul 08 19:30:09.096745 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:30:09.096727 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43484ea783e6f0738cb4850613b2f8ebeea7e49582df05ce7f1c01cf53bf0487\": container with ID starting with 43484ea783e6f0738cb4850613b2f8ebeea7e49582df05ce7f1c01cf53bf0487 not found: ID does not exist" containerID="43484ea783e6f0738cb4850613b2f8ebeea7e49582df05ce7f1c01cf53bf0487" Jul 08 19:30:09.096810 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:09.096752 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43484ea783e6f0738cb4850613b2f8ebeea7e49582df05ce7f1c01cf53bf0487"} err="failed to get container status \"43484ea783e6f0738cb4850613b2f8ebeea7e49582df05ce7f1c01cf53bf0487\": rpc error: code = NotFound desc = could not find container \"43484ea783e6f0738cb4850613b2f8ebeea7e49582df05ce7f1c01cf53bf0487\": container with ID starting with 43484ea783e6f0738cb4850613b2f8ebeea7e49582df05ce7f1c01cf53bf0487 not found: ID does not exist" Jul 08 19:30:09.114807 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:09.114781 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-19b7c553-8655fc775b-tsqgr"] Jul 08 19:30:09.121142 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:09.121119 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-19b7c553-8655fc775b-tsqgr"] Jul 08 19:30:09.282075 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:09.282038 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3271ae7b-4ca6-4975-836e-88906ac8a837" path="/var/lib/kubelet/pods/3271ae7b-4ca6-4975-836e-88906ac8a837/volumes" Jul 08 19:30:14.151669 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:14.151632 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:14.253685 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:14.253641 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:16.785067 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:16.785024 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:20.051896 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:20.051829 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:20.120377 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:20.120336 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:20.282431 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:20.282377 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:20.380432 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:20.380384 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:21.436184 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:21.436145 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:21.509365 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:21.509327 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:21.518144 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:21.518117 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f7c8045c-57948ff646-zbxvl"] Jul 08 19:30:21.518478 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:21.518466 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3271ae7b-4ca6-4975-836e-88906ac8a837" containerName="maas-api" Jul 08 19:30:21.518519 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:21.518480 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3271ae7b-4ca6-4975-836e-88906ac8a837" containerName="maas-api" Jul 08 19:30:21.518552 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:21.518543 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3271ae7b-4ca6-4975-836e-88906ac8a837" containerName="maas-api" Jul 08 19:30:21.522785 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:21.522768 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f7c8045c-57948ff646-zbxvl" Jul 08 19:30:21.525037 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:21.525012 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-f7c8045c-serving-cert\"" Jul 08 19:30:21.530288 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:21.530268 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f7c8045c-57948ff646-zbxvl"] Jul 08 19:30:21.585049 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:21.584995 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:21.628295 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:21.628262 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7939efaa-61b0-4c54-8fee-d08009bcf050-maas-api-tls\") pod \"maas-api-e2e-derive-f7c8045c-57948ff646-zbxvl\" (UID: \"7939efaa-61b0-4c54-8fee-d08009bcf050\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f7c8045c-57948ff646-zbxvl" Jul 08 19:30:21.628466 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:21.628350 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn4tw\" (UniqueName: \"kubernetes.io/projected/7939efaa-61b0-4c54-8fee-d08009bcf050-kube-api-access-nn4tw\") pod \"maas-api-e2e-derive-f7c8045c-57948ff646-zbxvl\" (UID: \"7939efaa-61b0-4c54-8fee-d08009bcf050\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f7c8045c-57948ff646-zbxvl" Jul 08 19:30:21.628466 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:21.628412 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7939efaa-61b0-4c54-8fee-d08009bcf050-openshift-service-ca\") pod \"maas-api-e2e-derive-f7c8045c-57948ff646-zbxvl\" (UID: \"7939efaa-61b0-4c54-8fee-d08009bcf050\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f7c8045c-57948ff646-zbxvl" Jul 08 19:30:21.728915 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:21.728833 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7939efaa-61b0-4c54-8fee-d08009bcf050-maas-api-tls\") pod \"maas-api-e2e-derive-f7c8045c-57948ff646-zbxvl\" (UID: \"7939efaa-61b0-4c54-8fee-d08009bcf050\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f7c8045c-57948ff646-zbxvl" Jul 08 19:30:21.728915 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:21.728892 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn4tw\" (UniqueName: \"kubernetes.io/projected/7939efaa-61b0-4c54-8fee-d08009bcf050-kube-api-access-nn4tw\") pod \"maas-api-e2e-derive-f7c8045c-57948ff646-zbxvl\" (UID: \"7939efaa-61b0-4c54-8fee-d08009bcf050\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f7c8045c-57948ff646-zbxvl" Jul 08 19:30:21.729109 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:21.728943 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7939efaa-61b0-4c54-8fee-d08009bcf050-openshift-service-ca\") pod \"maas-api-e2e-derive-f7c8045c-57948ff646-zbxvl\" (UID: \"7939efaa-61b0-4c54-8fee-d08009bcf050\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f7c8045c-57948ff646-zbxvl" Jul 08 19:30:21.729109 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:30:21.728986 2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-f7c8045c-serving-cert: secret "maas-api-e2e-derive-f7c8045c-serving-cert" not found Jul 08 19:30:21.729109 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:30:21.729070 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7939efaa-61b0-4c54-8fee-d08009bcf050-maas-api-tls podName:7939efaa-61b0-4c54-8fee-d08009bcf050 nodeName:}" failed. No retries permitted until 2026-07-08 19:30:22.229049362 +0000 UTC m=+1605.488002272 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7939efaa-61b0-4c54-8fee-d08009bcf050-maas-api-tls") pod "maas-api-e2e-derive-f7c8045c-57948ff646-zbxvl" (UID: "7939efaa-61b0-4c54-8fee-d08009bcf050") : secret "maas-api-e2e-derive-f7c8045c-serving-cert" not found Jul 08 19:30:21.729556 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:21.729535 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7939efaa-61b0-4c54-8fee-d08009bcf050-openshift-service-ca\") pod \"maas-api-e2e-derive-f7c8045c-57948ff646-zbxvl\" (UID: \"7939efaa-61b0-4c54-8fee-d08009bcf050\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f7c8045c-57948ff646-zbxvl" Jul 08 19:30:21.738280 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:21.738246 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn4tw\" (UniqueName: \"kubernetes.io/projected/7939efaa-61b0-4c54-8fee-d08009bcf050-kube-api-access-nn4tw\") pod \"maas-api-e2e-derive-f7c8045c-57948ff646-zbxvl\" (UID: \"7939efaa-61b0-4c54-8fee-d08009bcf050\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f7c8045c-57948ff646-zbxvl" Jul 08 19:30:22.025208 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:22.025113 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:22.168369 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:22.168326 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:22.234142 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:22.234108 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7939efaa-61b0-4c54-8fee-d08009bcf050-maas-api-tls\") pod \"maas-api-e2e-derive-f7c8045c-57948ff646-zbxvl\" (UID: \"7939efaa-61b0-4c54-8fee-d08009bcf050\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f7c8045c-57948ff646-zbxvl" Jul 08 19:30:22.236757 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:22.236724 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7939efaa-61b0-4c54-8fee-d08009bcf050-maas-api-tls\") pod \"maas-api-e2e-derive-f7c8045c-57948ff646-zbxvl\" (UID: \"7939efaa-61b0-4c54-8fee-d08009bcf050\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f7c8045c-57948ff646-zbxvl" Jul 08 19:30:22.266446 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:22.266410 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:22.434965 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:22.434929 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f7c8045c-57948ff646-zbxvl" Jul 08 19:30:22.577644 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:22.577615 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f7c8045c-57948ff646-zbxvl"] Jul 08 19:30:22.579531 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:30:22.579499 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7939efaa_61b0_4c54_8fee_d08009bcf050.slice/crio-6f98432e55d233d0bcf3f15a357bbda5bb0a1b8a2d67f3372e1099bbdc1f98ee WatchSource:0}: Error finding container 6f98432e55d233d0bcf3f15a357bbda5bb0a1b8a2d67f3372e1099bbdc1f98ee: Status 404 returned error can't find the container with id 6f98432e55d233d0bcf3f15a357bbda5bb0a1b8a2d67f3372e1099bbdc1f98ee Jul 08 19:30:23.159818 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:23.159771 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f7c8045c-57948ff646-zbxvl" event={"ID":"7939efaa-61b0-4c54-8fee-d08009bcf050","Type":"ContainerStarted","Data":"ca088b1b82ad4fe92ccfd567b78eeefe0e4680e85a35657556d7d0f0bd9931e9"} Jul 08 19:30:23.159972 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:23.159869 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f7c8045c-57948ff646-zbxvl" event={"ID":"7939efaa-61b0-4c54-8fee-d08009bcf050","Type":"ContainerStarted","Data":"6f98432e55d233d0bcf3f15a357bbda5bb0a1b8a2d67f3372e1099bbdc1f98ee"} Jul 08 19:30:23.159972 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:23.159919 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f7c8045c-57948ff646-zbxvl" Jul 08 19:30:23.179741 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:23.179691 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f7c8045c-57948ff646-zbxvl" podStartSLOduration=1.812264774 podStartE2EDuration="2.17967604s" totalImagesPullingTime="367.411266ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:30:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:30:23.178612341 +0000 UTC m=+1606.437565257" watchObservedRunningTime="2026-07-08 19:30:23.17967604 +0000 UTC m=+1606.438628965" Jul 08 19:30:23.402755 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:23.402721 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:25.796788 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:25.796751 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:25.998380 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:25.998339 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:26.130934 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:26.130902 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f7c8045c-57948ff646-zbxvl"] Jul 08 19:30:26.131172 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:26.131142 2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f7c8045c-57948ff646-zbxvl" podUID="7939efaa-61b0-4c54-8fee-d08009bcf050" containerName="maas-api" containerID="cri-o://ca088b1b82ad4fe92ccfd567b78eeefe0e4680e85a35657556d7d0f0bd9931e9" gracePeriod=30 Jul 08 19:30:26.356678 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:26.356630 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:26.380995 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:26.380941 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f7c8045c-57948ff646-zbxvl" Jul 08 19:30:26.473301 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:26.473244 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:26.477205 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:26.477186 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nn4tw\" (UniqueName: \"kubernetes.io/projected/7939efaa-61b0-4c54-8fee-d08009bcf050-kube-api-access-nn4tw\") pod \"7939efaa-61b0-4c54-8fee-d08009bcf050\" (UID: \"7939efaa-61b0-4c54-8fee-d08009bcf050\") " Jul 08 19:30:26.477282 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:26.477241 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7939efaa-61b0-4c54-8fee-d08009bcf050-openshift-service-ca\") pod \"7939efaa-61b0-4c54-8fee-d08009bcf050\" (UID: \"7939efaa-61b0-4c54-8fee-d08009bcf050\") " Jul 08 19:30:26.477319 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:26.477279 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7939efaa-61b0-4c54-8fee-d08009bcf050-maas-api-tls\") pod \"7939efaa-61b0-4c54-8fee-d08009bcf050\" (UID: \"7939efaa-61b0-4c54-8fee-d08009bcf050\") " Jul 08 19:30:26.477670 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:26.477648 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7939efaa-61b0-4c54-8fee-d08009bcf050-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7939efaa-61b0-4c54-8fee-d08009bcf050" (UID: "7939efaa-61b0-4c54-8fee-d08009bcf050"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:30:26.479326 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:26.479302 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7939efaa-61b0-4c54-8fee-d08009bcf050-kube-api-access-nn4tw" (OuterVolumeSpecName: "kube-api-access-nn4tw") pod "7939efaa-61b0-4c54-8fee-d08009bcf050" (UID: "7939efaa-61b0-4c54-8fee-d08009bcf050"). InnerVolumeSpecName "kube-api-access-nn4tw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:30:26.479447 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:26.479331 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7939efaa-61b0-4c54-8fee-d08009bcf050-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7939efaa-61b0-4c54-8fee-d08009bcf050" (UID: "7939efaa-61b0-4c54-8fee-d08009bcf050"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:30:26.578712 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:26.578677 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7939efaa-61b0-4c54-8fee-d08009bcf050-maas-api-tls\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:30:26.578712 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:26.578706 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn4tw\" (UniqueName: \"kubernetes.io/projected/7939efaa-61b0-4c54-8fee-d08009bcf050-kube-api-access-nn4tw\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:30:26.578712 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:26.578716 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7939efaa-61b0-4c54-8fee-d08009bcf050-openshift-service-ca\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:30:27.180247 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:27.180214 2558 generic.go:356] "Generic (PLEG): container finished" podID="7939efaa-61b0-4c54-8fee-d08009bcf050" containerID="ca088b1b82ad4fe92ccfd567b78eeefe0e4680e85a35657556d7d0f0bd9931e9" exitCode=0 Jul 08 19:30:27.180716 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:27.180259 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f7c8045c-57948ff646-zbxvl" Jul 08 19:30:27.180716 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:27.180284 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f7c8045c-57948ff646-zbxvl" event={"ID":"7939efaa-61b0-4c54-8fee-d08009bcf050","Type":"ContainerDied","Data":"ca088b1b82ad4fe92ccfd567b78eeefe0e4680e85a35657556d7d0f0bd9931e9"} Jul 08 19:30:27.180716 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:27.180326 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f7c8045c-57948ff646-zbxvl" event={"ID":"7939efaa-61b0-4c54-8fee-d08009bcf050","Type":"ContainerDied","Data":"6f98432e55d233d0bcf3f15a357bbda5bb0a1b8a2d67f3372e1099bbdc1f98ee"} Jul 08 19:30:27.180716 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:27.180351 2558 scope.go:117] "RemoveContainer" containerID="ca088b1b82ad4fe92ccfd567b78eeefe0e4680e85a35657556d7d0f0bd9931e9" Jul 08 19:30:27.191459 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:27.191433 2558 scope.go:117] "RemoveContainer" containerID="ca088b1b82ad4fe92ccfd567b78eeefe0e4680e85a35657556d7d0f0bd9931e9" Jul 08 19:30:27.192023 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:30:27.192004 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca088b1b82ad4fe92ccfd567b78eeefe0e4680e85a35657556d7d0f0bd9931e9\": container with ID starting with ca088b1b82ad4fe92ccfd567b78eeefe0e4680e85a35657556d7d0f0bd9931e9 not found: ID does not exist" containerID="ca088b1b82ad4fe92ccfd567b78eeefe0e4680e85a35657556d7d0f0bd9931e9" Jul 08 19:30:27.192112 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:27.192032 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca088b1b82ad4fe92ccfd567b78eeefe0e4680e85a35657556d7d0f0bd9931e9"} err="failed to get container status \"ca088b1b82ad4fe92ccfd567b78eeefe0e4680e85a35657556d7d0f0bd9931e9\": rpc error: code = NotFound desc = could not find container \"ca088b1b82ad4fe92ccfd567b78eeefe0e4680e85a35657556d7d0f0bd9931e9\": container with ID starting with ca088b1b82ad4fe92ccfd567b78eeefe0e4680e85a35657556d7d0f0bd9931e9 not found: ID does not exist" Jul 08 19:30:27.202138 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:27.202117 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f7c8045c-57948ff646-zbxvl"] Jul 08 19:30:27.206183 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:27.206162 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f7c8045c-57948ff646-zbxvl"] Jul 08 19:30:27.282097 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:27.282064 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7939efaa-61b0-4c54-8fee-d08009bcf050" path="/var/lib/kubelet/pods/7939efaa-61b0-4c54-8fee-d08009bcf050/volumes" Jul 08 19:30:27.598764 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:27.598725 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:31.960707 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:31.960609 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:32.017149 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:32.017111 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:32.425394 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:32.425361 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:32.521387 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:32.521350 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:32.620994 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:32.620955 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:33.944135 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:33.944101 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:39.940694 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:39.940662 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:40.040097 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:40.040056 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:40.140335 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:40.140294 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:40.239385 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:40.239297 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:30:40.652199 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:30:40.652173 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:31:00.011199 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:31:00.011154 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725650-tctpl"] Jul 08 19:31:00.011766 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:31:00.011431 2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725650-tctpl" podUID="985a70ac-d361-4e9e-afbb-af65bd49d584" containerName="cleanup" containerID="cri-o://a44de3b47b439555201507f9ab1f139a30d42a41b40b8d4a09caea1c924a6a2e" gracePeriod=30 Jul 08 19:31:06.652544 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:31:06.652509 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:31:12.638633 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:31:12.638587 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:31:12.700525 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:31:12.700482 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:31:12.763897 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:31:12.763856 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:31:12.862396 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:31:12.862361 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:31:22.638701 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:31:22.638644 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:31:22.694154 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:31:22.694120 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:31:28.732063 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:31:28.732030 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:31:30.202267 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:31:30.202238 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725650-tctpl" Jul 08 19:31:30.318606 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:31:30.318503 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2cm9t\" (UniqueName: \"kubernetes.io/projected/985a70ac-d361-4e9e-afbb-af65bd49d584-kube-api-access-2cm9t\") pod \"985a70ac-d361-4e9e-afbb-af65bd49d584\" (UID: \"985a70ac-d361-4e9e-afbb-af65bd49d584\") " Jul 08 19:31:30.320535 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:31:30.320509 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/985a70ac-d361-4e9e-afbb-af65bd49d584-kube-api-access-2cm9t" (OuterVolumeSpecName: "kube-api-access-2cm9t") pod "985a70ac-d361-4e9e-afbb-af65bd49d584" (UID: "985a70ac-d361-4e9e-afbb-af65bd49d584"). InnerVolumeSpecName "kube-api-access-2cm9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:31:30.419442 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:31:30.419414 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cm9t\" (UniqueName: \"kubernetes.io/projected/985a70ac-d361-4e9e-afbb-af65bd49d584-kube-api-access-2cm9t\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:31:30.498243 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:31:30.498206 2558 generic.go:356] "Generic (PLEG): container finished" podID="985a70ac-d361-4e9e-afbb-af65bd49d584" containerID="a44de3b47b439555201507f9ab1f139a30d42a41b40b8d4a09caea1c924a6a2e" exitCode=137 Jul 08 19:31:30.498400 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:31:30.498257 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725650-tctpl" Jul 08 19:31:30.498400 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:31:30.498290 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725650-tctpl" event={"ID":"985a70ac-d361-4e9e-afbb-af65bd49d584","Type":"ContainerDied","Data":"a44de3b47b439555201507f9ab1f139a30d42a41b40b8d4a09caea1c924a6a2e"} Jul 08 19:31:30.498400 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:31:30.498348 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725650-tctpl" event={"ID":"985a70ac-d361-4e9e-afbb-af65bd49d584","Type":"ContainerDied","Data":"8cc467f506ad4350dca6c0cbfc4a28a076c7504a9872b28b0cdd1bbd739faa7d"} Jul 08 19:31:30.498400 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:31:30.498370 2558 scope.go:117] "RemoveContainer" containerID="a44de3b47b439555201507f9ab1f139a30d42a41b40b8d4a09caea1c924a6a2e" Jul 08 19:31:30.507851 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:31:30.507836 2558 scope.go:117] "RemoveContainer" containerID="a44de3b47b439555201507f9ab1f139a30d42a41b40b8d4a09caea1c924a6a2e" Jul 08 19:31:30.508063 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:31:30.508046 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a44de3b47b439555201507f9ab1f139a30d42a41b40b8d4a09caea1c924a6a2e\": container with ID starting with a44de3b47b439555201507f9ab1f139a30d42a41b40b8d4a09caea1c924a6a2e not found: ID does not exist" containerID="a44de3b47b439555201507f9ab1f139a30d42a41b40b8d4a09caea1c924a6a2e" Jul 08 19:31:30.508111 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:31:30.508070 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a44de3b47b439555201507f9ab1f139a30d42a41b40b8d4a09caea1c924a6a2e"} err="failed to get container status \"a44de3b47b439555201507f9ab1f139a30d42a41b40b8d4a09caea1c924a6a2e\": rpc error: code = NotFound desc = could not find container \"a44de3b47b439555201507f9ab1f139a30d42a41b40b8d4a09caea1c924a6a2e\": container with ID starting with a44de3b47b439555201507f9ab1f139a30d42a41b40b8d4a09caea1c924a6a2e not found: ID does not exist" Jul 08 19:31:30.521953 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:31:30.521075 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725650-tctpl"] Jul 08 19:31:30.528409 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:31:30.528385 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725650-tctpl"] Jul 08 19:31:31.282084 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:31:31.282051 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="985a70ac-d361-4e9e-afbb-af65bd49d584" path="/var/lib/kubelet/pods/985a70ac-d361-4e9e-afbb-af65bd49d584/volumes" Jul 08 19:31:34.650786 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:31:34.650749 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:31:34.756599 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:31:34.756539 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:31:34.849463 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:31:34.849346 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:31:44.136253 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:31:44.136221 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:31:45.371178 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:31:45.371147 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:31:46.216156 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:31:46.216119 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:31:52.626856 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:31:52.626819 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:31:52.700179 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:31:52.700142 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:31:52.774662 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:31:52.774610 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:31:52.874308 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:31:52.874271 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:31:52.982367 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:31:52.982279 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:31:53.798376 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:31:53.798343 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:31:53.938598 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:31:53.938524 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:31:54.019806 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:31:54.019774 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:31:55.604460 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:31:55.604413 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:01.449882 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:01.449792 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:01.504620 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:01.504562 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:02.132011 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:02.131974 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:08.104086 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:08.104052 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:08.166589 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:08.166551 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:08.343894 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:08.343856 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:08.648595 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:08.648543 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:08.910251 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:08.910164 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:11.568363 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:11.568319 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:11.654180 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:11.654134 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:11.655415 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:11.655393 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-94233cba-54c6564979-2blv9"] Jul 08 19:32:11.655805 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:11.655788 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="985a70ac-d361-4e9e-afbb-af65bd49d584" containerName="cleanup" Jul 08 19:32:11.655805 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:11.655805 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="985a70ac-d361-4e9e-afbb-af65bd49d584" containerName="cleanup" Jul 08 19:32:11.655941 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:11.655817 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7939efaa-61b0-4c54-8fee-d08009bcf050" containerName="maas-api" Jul 08 19:32:11.655941 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:11.655827 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7939efaa-61b0-4c54-8fee-d08009bcf050" containerName="maas-api" Jul 08 19:32:11.655941 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:11.655900 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7939efaa-61b0-4c54-8fee-d08009bcf050" containerName="maas-api" Jul 08 19:32:11.655941 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:11.655912 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="985a70ac-d361-4e9e-afbb-af65bd49d584" containerName="cleanup" Jul 08 19:32:11.659000 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:11.658977 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-94233cba-54c6564979-2blv9" Jul 08 19:32:11.661361 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:11.661336 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-94233cba-serving-cert\"" Jul 08 19:32:11.668625 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:11.668602 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-94233cba-54c6564979-2blv9"] Jul 08 19:32:11.682927 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:11.682902 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc4fc38c-fc70-4437-b5b7-4118c2352b57-openshift-service-ca\") pod \"maas-api-e2e-mt-94233cba-54c6564979-2blv9\" (UID: \"dc4fc38c-fc70-4437-b5b7-4118c2352b57\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-94233cba-54c6564979-2blv9" Jul 08 19:32:11.683021 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:11.682939 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brs2r\" (UniqueName: \"kubernetes.io/projected/dc4fc38c-fc70-4437-b5b7-4118c2352b57-kube-api-access-brs2r\") pod \"maas-api-e2e-mt-94233cba-54c6564979-2blv9\" (UID: \"dc4fc38c-fc70-4437-b5b7-4118c2352b57\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-94233cba-54c6564979-2blv9" Jul 08 19:32:11.683078 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:11.683029 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc4fc38c-fc70-4437-b5b7-4118c2352b57-maas-api-tls\") pod \"maas-api-e2e-mt-94233cba-54c6564979-2blv9\" (UID: \"dc4fc38c-fc70-4437-b5b7-4118c2352b57\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-94233cba-54c6564979-2blv9" Jul 08 19:32:11.718128 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:11.718095 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:11.783906 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:11.783879 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc4fc38c-fc70-4437-b5b7-4118c2352b57-maas-api-tls\") pod \"maas-api-e2e-mt-94233cba-54c6564979-2blv9\" (UID: \"dc4fc38c-fc70-4437-b5b7-4118c2352b57\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-94233cba-54c6564979-2blv9" Jul 08 19:32:11.784109 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:11.783928 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc4fc38c-fc70-4437-b5b7-4118c2352b57-openshift-service-ca\") pod \"maas-api-e2e-mt-94233cba-54c6564979-2blv9\" (UID: \"dc4fc38c-fc70-4437-b5b7-4118c2352b57\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-94233cba-54c6564979-2blv9" Jul 08 19:32:11.784109 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:11.783955 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brs2r\" (UniqueName: \"kubernetes.io/projected/dc4fc38c-fc70-4437-b5b7-4118c2352b57-kube-api-access-brs2r\") pod \"maas-api-e2e-mt-94233cba-54c6564979-2blv9\" (UID: \"dc4fc38c-fc70-4437-b5b7-4118c2352b57\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-94233cba-54c6564979-2blv9" Jul 08 19:32:11.784109 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:32:11.784027 2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-94233cba-serving-cert: secret "maas-api-e2e-mt-94233cba-serving-cert" not found Jul 08 19:32:11.784109 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:32:11.784105 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc4fc38c-fc70-4437-b5b7-4118c2352b57-maas-api-tls podName:dc4fc38c-fc70-4437-b5b7-4118c2352b57 nodeName:}" failed. No retries permitted until 2026-07-08 19:32:12.284085527 +0000 UTC m=+1715.543038444 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dc4fc38c-fc70-4437-b5b7-4118c2352b57-maas-api-tls") pod "maas-api-e2e-mt-94233cba-54c6564979-2blv9" (UID: "dc4fc38c-fc70-4437-b5b7-4118c2352b57") : secret "maas-api-e2e-mt-94233cba-serving-cert" not found Jul 08 19:32:11.784657 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:11.784635 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc4fc38c-fc70-4437-b5b7-4118c2352b57-openshift-service-ca\") pod \"maas-api-e2e-mt-94233cba-54c6564979-2blv9\" (UID: \"dc4fc38c-fc70-4437-b5b7-4118c2352b57\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-94233cba-54c6564979-2blv9" Jul 08 19:32:11.793198 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:11.793173 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brs2r\" (UniqueName: \"kubernetes.io/projected/dc4fc38c-fc70-4437-b5b7-4118c2352b57-kube-api-access-brs2r\") pod \"maas-api-e2e-mt-94233cba-54c6564979-2blv9\" (UID: \"dc4fc38c-fc70-4437-b5b7-4118c2352b57\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-94233cba-54c6564979-2blv9" Jul 08 19:32:12.286593 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:12.286549 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:12.288694 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:12.288667 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc4fc38c-fc70-4437-b5b7-4118c2352b57-maas-api-tls\") pod \"maas-api-e2e-mt-94233cba-54c6564979-2blv9\" (UID: \"dc4fc38c-fc70-4437-b5b7-4118c2352b57\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-94233cba-54c6564979-2blv9" Jul 08 19:32:12.288811 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:32:12.288795 2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-94233cba-serving-cert: secret "maas-api-e2e-mt-94233cba-serving-cert" not found Jul 08 19:32:12.288868 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:32:12.288858 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc4fc38c-fc70-4437-b5b7-4118c2352b57-maas-api-tls podName:dc4fc38c-fc70-4437-b5b7-4118c2352b57 nodeName:}" failed. No retries permitted until 2026-07-08 19:32:13.288843341 +0000 UTC m=+1716.547796244 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dc4fc38c-fc70-4437-b5b7-4118c2352b57-maas-api-tls") pod "maas-api-e2e-mt-94233cba-54c6564979-2blv9" (UID: "dc4fc38c-fc70-4437-b5b7-4118c2352b57") : secret "maas-api-e2e-mt-94233cba-serving-cert" not found Jul 08 19:32:12.383706 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:12.383671 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:12.484108 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:12.484064 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:13.299906 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:13.299874 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc4fc38c-fc70-4437-b5b7-4118c2352b57-maas-api-tls\") pod \"maas-api-e2e-mt-94233cba-54c6564979-2blv9\" (UID: \"dc4fc38c-fc70-4437-b5b7-4118c2352b57\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-94233cba-54c6564979-2blv9" Jul 08 19:32:13.302342 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:13.302316 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc4fc38c-fc70-4437-b5b7-4118c2352b57-maas-api-tls\") pod \"maas-api-e2e-mt-94233cba-54c6564979-2blv9\" (UID: \"dc4fc38c-fc70-4437-b5b7-4118c2352b57\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-94233cba-54c6564979-2blv9" Jul 08 19:32:13.471637 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:13.471561 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-94233cba-54c6564979-2blv9" Jul 08 19:32:13.613495 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:13.613468 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-94233cba-54c6564979-2blv9"] Jul 08 19:32:13.615735 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:32:13.615702 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc4fc38c_fc70_4437_b5b7_4118c2352b57.slice/crio-8d1011cb0b1d0f194e0e53dfab43991b6c3436a0f8ab9fa662d7009b0027c53b WatchSource:0}: Error finding container 8d1011cb0b1d0f194e0e53dfab43991b6c3436a0f8ab9fa662d7009b0027c53b: Status 404 returned error can't find the container with id 8d1011cb0b1d0f194e0e53dfab43991b6c3436a0f8ab9fa662d7009b0027c53b Jul 08 19:32:13.721958 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:13.721924 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-94233cba-54c6564979-2blv9" event={"ID":"dc4fc38c-fc70-4437-b5b7-4118c2352b57","Type":"ContainerStarted","Data":"8d1011cb0b1d0f194e0e53dfab43991b6c3436a0f8ab9fa662d7009b0027c53b"} Jul 08 19:32:14.728829 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:14.728797 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-94233cba-54c6564979-2blv9" event={"ID":"dc4fc38c-fc70-4437-b5b7-4118c2352b57","Type":"ContainerStarted","Data":"896bdec40bb9be019c9afcf66ff50f823782fa4fecef2b1d3743027e7c06815b"} Jul 08 19:32:14.729207 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:14.728894 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-94233cba-54c6564979-2blv9" Jul 08 19:32:14.747589 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:14.747511 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-94233cba-54c6564979-2blv9" podStartSLOduration=3.367068452 podStartE2EDuration="3.747495162s" totalImagesPullingTime="380.42671ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:32:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:32:14.746136945 +0000 UTC m=+1718.005089911" watchObservedRunningTime="2026-07-08 19:32:14.747495162 +0000 UTC m=+1718.006448064" Jul 08 19:32:16.076175 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:16.076137 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:16.734436 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:16.734400 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:16.826630 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:16.826600 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:17.204973 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:17.204934 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:20.739653 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:20.739624 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-94233cba-54c6564979-2blv9" Jul 08 19:32:23.297818 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:23.297776 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:23.370199 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:23.370162 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:23.703924 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:23.703885 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:24.018767 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:24.018676 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-94233cba-54c6564979-2blv9"] Jul 08 19:32:24.018966 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:24.018931 2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-94233cba-54c6564979-2blv9" podUID="dc4fc38c-fc70-4437-b5b7-4118c2352b57" containerName="maas-api" containerID="cri-o://896bdec40bb9be019c9afcf66ff50f823782fa4fecef2b1d3743027e7c06815b" gracePeriod=30 Jul 08 19:32:24.065072 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:24.065040 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:24.138426 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:24.138373 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:24.270854 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:24.270794 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-94233cba-54c6564979-2blv9" Jul 08 19:32:24.392291 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:24.392260 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc4fc38c-fc70-4437-b5b7-4118c2352b57-openshift-service-ca\") pod \"dc4fc38c-fc70-4437-b5b7-4118c2352b57\" (UID: \"dc4fc38c-fc70-4437-b5b7-4118c2352b57\") " Jul 08 19:32:24.392778 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:24.392312 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc4fc38c-fc70-4437-b5b7-4118c2352b57-maas-api-tls\") pod \"dc4fc38c-fc70-4437-b5b7-4118c2352b57\" (UID: \"dc4fc38c-fc70-4437-b5b7-4118c2352b57\") " Jul 08 19:32:24.392778 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:24.392362 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brs2r\" (UniqueName: \"kubernetes.io/projected/dc4fc38c-fc70-4437-b5b7-4118c2352b57-kube-api-access-brs2r\") pod \"dc4fc38c-fc70-4437-b5b7-4118c2352b57\" (UID: \"dc4fc38c-fc70-4437-b5b7-4118c2352b57\") " Jul 08 19:32:24.392778 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:24.392725 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc4fc38c-fc70-4437-b5b7-4118c2352b57-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dc4fc38c-fc70-4437-b5b7-4118c2352b57" (UID: "dc4fc38c-fc70-4437-b5b7-4118c2352b57"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:32:24.394419 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:24.394397 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc4fc38c-fc70-4437-b5b7-4118c2352b57-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dc4fc38c-fc70-4437-b5b7-4118c2352b57" (UID: "dc4fc38c-fc70-4437-b5b7-4118c2352b57"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:32:24.394496 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:24.394458 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc4fc38c-fc70-4437-b5b7-4118c2352b57-kube-api-access-brs2r" (OuterVolumeSpecName: "kube-api-access-brs2r") pod "dc4fc38c-fc70-4437-b5b7-4118c2352b57" (UID: "dc4fc38c-fc70-4437-b5b7-4118c2352b57"). InnerVolumeSpecName "kube-api-access-brs2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:32:24.493444 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:24.493415 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc4fc38c-fc70-4437-b5b7-4118c2352b57-maas-api-tls\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:32:24.493444 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:24.493440 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brs2r\" (UniqueName: \"kubernetes.io/projected/dc4fc38c-fc70-4437-b5b7-4118c2352b57-kube-api-access-brs2r\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:32:24.493444 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:24.493450 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc4fc38c-fc70-4437-b5b7-4118c2352b57-openshift-service-ca\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:32:24.784882 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:24.784846 2558 generic.go:356] "Generic (PLEG): container finished" podID="dc4fc38c-fc70-4437-b5b7-4118c2352b57" containerID="896bdec40bb9be019c9afcf66ff50f823782fa4fecef2b1d3743027e7c06815b" exitCode=0 Jul 08 19:32:24.785054 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:24.784895 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-94233cba-54c6564979-2blv9" Jul 08 19:32:24.785054 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:24.784938 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-94233cba-54c6564979-2blv9" event={"ID":"dc4fc38c-fc70-4437-b5b7-4118c2352b57","Type":"ContainerDied","Data":"896bdec40bb9be019c9afcf66ff50f823782fa4fecef2b1d3743027e7c06815b"} Jul 08 19:32:24.785054 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:24.785005 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-94233cba-54c6564979-2blv9" event={"ID":"dc4fc38c-fc70-4437-b5b7-4118c2352b57","Type":"ContainerDied","Data":"8d1011cb0b1d0f194e0e53dfab43991b6c3436a0f8ab9fa662d7009b0027c53b"} Jul 08 19:32:24.785054 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:24.785031 2558 scope.go:117] "RemoveContainer" containerID="896bdec40bb9be019c9afcf66ff50f823782fa4fecef2b1d3743027e7c06815b" Jul 08 19:32:24.794603 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:24.794568 2558 scope.go:117] "RemoveContainer" containerID="896bdec40bb9be019c9afcf66ff50f823782fa4fecef2b1d3743027e7c06815b" Jul 08 19:32:24.794906 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:32:24.794876 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"896bdec40bb9be019c9afcf66ff50f823782fa4fecef2b1d3743027e7c06815b\": container with ID starting with 896bdec40bb9be019c9afcf66ff50f823782fa4fecef2b1d3743027e7c06815b not found: ID does not exist" containerID="896bdec40bb9be019c9afcf66ff50f823782fa4fecef2b1d3743027e7c06815b" Jul 08 19:32:24.794969 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:24.794921 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"896bdec40bb9be019c9afcf66ff50f823782fa4fecef2b1d3743027e7c06815b"} err="failed to get container status \"896bdec40bb9be019c9afcf66ff50f823782fa4fecef2b1d3743027e7c06815b\": rpc error: code = NotFound desc = could not find container \"896bdec40bb9be019c9afcf66ff50f823782fa4fecef2b1d3743027e7c06815b\": container with ID starting with 896bdec40bb9be019c9afcf66ff50f823782fa4fecef2b1d3743027e7c06815b not found: ID does not exist" Jul 08 19:32:24.808986 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:24.808957 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-94233cba-54c6564979-2blv9"] Jul 08 19:32:24.814338 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:24.814315 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-94233cba-54c6564979-2blv9"] Jul 08 19:32:25.283452 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:25.283419 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc4fc38c-fc70-4437-b5b7-4118c2352b57" path="/var/lib/kubelet/pods/dc4fc38c-fc70-4437-b5b7-4118c2352b57/volumes" Jul 08 19:32:30.171168 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:30.171130 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:30.243748 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:30.243711 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:30.432736 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:30.432656 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:30.774738 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:30.774638 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:30.873182 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:30.873131 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:30.989672 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:30.989637 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:37.254278 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:37.254239 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:37.314858 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:37.314822 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:37.392194 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:37.392161 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:37.488953 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:37.488895 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:37.608495 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:37.608402 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:43.684712 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:43.684671 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:43.756501 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:43.756468 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:47.012232 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:47.012189 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:47.112820 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:47.112780 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:50.578907 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:50.578866 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:50.884978 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:50.884944 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:51.800982 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:51.800939 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:53.781611 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:53.781526 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:53.877347 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:53.877311 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:53.902676 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:53.902642 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7c480b69-5456799f87-8s6pm"] Jul 08 19:32:53.903065 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:53.903047 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc4fc38c-fc70-4437-b5b7-4118c2352b57" containerName="maas-api" Jul 08 19:32:53.903113 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:53.903067 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc4fc38c-fc70-4437-b5b7-4118c2352b57" containerName="maas-api" Jul 08 19:32:53.903148 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:53.903143 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc4fc38c-fc70-4437-b5b7-4118c2352b57" containerName="maas-api" Jul 08 19:32:53.906270 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:53.906255 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c480b69-5456799f87-8s6pm" Jul 08 19:32:53.908717 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:53.908685 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-7c480b69-serving-cert\"" Jul 08 19:32:53.916260 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:53.916240 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7c480b69-5456799f87-8s6pm"] Jul 08 19:32:53.944519 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:53.944484 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbk8z\" (UniqueName: \"kubernetes.io/projected/058f0a1a-31c0-4f1f-8080-0f80df5a9419-kube-api-access-zbk8z\") pod \"maas-api-e2e-mt-7c480b69-5456799f87-8s6pm\" (UID: \"058f0a1a-31c0-4f1f-8080-0f80df5a9419\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c480b69-5456799f87-8s6pm" Jul 08 19:32:53.944704 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:53.944547 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/058f0a1a-31c0-4f1f-8080-0f80df5a9419-openshift-service-ca\") pod \"maas-api-e2e-mt-7c480b69-5456799f87-8s6pm\" (UID: \"058f0a1a-31c0-4f1f-8080-0f80df5a9419\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c480b69-5456799f87-8s6pm" Jul 08 19:32:53.944704 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:53.944602 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/058f0a1a-31c0-4f1f-8080-0f80df5a9419-maas-api-tls\") pod \"maas-api-e2e-mt-7c480b69-5456799f87-8s6pm\" (UID: \"058f0a1a-31c0-4f1f-8080-0f80df5a9419\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c480b69-5456799f87-8s6pm" Jul 08 19:32:54.014695 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:54.014654 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:54.045742 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:54.045665 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbk8z\" (UniqueName: \"kubernetes.io/projected/058f0a1a-31c0-4f1f-8080-0f80df5a9419-kube-api-access-zbk8z\") pod \"maas-api-e2e-mt-7c480b69-5456799f87-8s6pm\" (UID: \"058f0a1a-31c0-4f1f-8080-0f80df5a9419\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c480b69-5456799f87-8s6pm" Jul 08 19:32:54.045742 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:54.045717 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/058f0a1a-31c0-4f1f-8080-0f80df5a9419-openshift-service-ca\") pod \"maas-api-e2e-mt-7c480b69-5456799f87-8s6pm\" (UID: \"058f0a1a-31c0-4f1f-8080-0f80df5a9419\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c480b69-5456799f87-8s6pm" Jul 08 19:32:54.045742 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:54.045740 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/058f0a1a-31c0-4f1f-8080-0f80df5a9419-maas-api-tls\") pod \"maas-api-e2e-mt-7c480b69-5456799f87-8s6pm\" (UID: \"058f0a1a-31c0-4f1f-8080-0f80df5a9419\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c480b69-5456799f87-8s6pm" Jul 08 19:32:54.045996 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:32:54.045866 2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-7c480b69-serving-cert: secret "maas-api-e2e-mt-7c480b69-serving-cert" not found Jul 08 19:32:54.045996 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:32:54.045941 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/058f0a1a-31c0-4f1f-8080-0f80df5a9419-maas-api-tls podName:058f0a1a-31c0-4f1f-8080-0f80df5a9419 nodeName:}" failed. No retries permitted until 2026-07-08 19:32:54.54592464 +0000 UTC m=+1757.804877542 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/058f0a1a-31c0-4f1f-8080-0f80df5a9419-maas-api-tls") pod "maas-api-e2e-mt-7c480b69-5456799f87-8s6pm" (UID: "058f0a1a-31c0-4f1f-8080-0f80df5a9419") : secret "maas-api-e2e-mt-7c480b69-serving-cert" not found Jul 08 19:32:54.046358 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:54.046340 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/058f0a1a-31c0-4f1f-8080-0f80df5a9419-openshift-service-ca\") pod \"maas-api-e2e-mt-7c480b69-5456799f87-8s6pm\" (UID: \"058f0a1a-31c0-4f1f-8080-0f80df5a9419\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c480b69-5456799f87-8s6pm" Jul 08 19:32:54.057668 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:54.057644 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbk8z\" (UniqueName: \"kubernetes.io/projected/058f0a1a-31c0-4f1f-8080-0f80df5a9419-kube-api-access-zbk8z\") pod \"maas-api-e2e-mt-7c480b69-5456799f87-8s6pm\" (UID: \"058f0a1a-31c0-4f1f-8080-0f80df5a9419\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c480b69-5456799f87-8s6pm" Jul 08 19:32:54.431462 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:54.431427 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:54.532474 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:54.532434 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:54.549770 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:54.549739 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/058f0a1a-31c0-4f1f-8080-0f80df5a9419-maas-api-tls\") pod \"maas-api-e2e-mt-7c480b69-5456799f87-8s6pm\" (UID: \"058f0a1a-31c0-4f1f-8080-0f80df5a9419\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c480b69-5456799f87-8s6pm" Jul 08 19:32:54.549878 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:32:54.549857 2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-7c480b69-serving-cert: secret "maas-api-e2e-mt-7c480b69-serving-cert" not found Jul 08 19:32:54.549931 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:32:54.549917 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/058f0a1a-31c0-4f1f-8080-0f80df5a9419-maas-api-tls podName:058f0a1a-31c0-4f1f-8080-0f80df5a9419 nodeName:}" failed. No retries permitted until 2026-07-08 19:32:55.549903629 +0000 UTC m=+1758.808856533 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/058f0a1a-31c0-4f1f-8080-0f80df5a9419-maas-api-tls") pod "maas-api-e2e-mt-7c480b69-5456799f87-8s6pm" (UID: "058f0a1a-31c0-4f1f-8080-0f80df5a9419") : secret "maas-api-e2e-mt-7c480b69-serving-cert" not found Jul 08 19:32:54.760166 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:54.760087 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:32:55.558377 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:55.558342 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/058f0a1a-31c0-4f1f-8080-0f80df5a9419-maas-api-tls\") pod \"maas-api-e2e-mt-7c480b69-5456799f87-8s6pm\" (UID: \"058f0a1a-31c0-4f1f-8080-0f80df5a9419\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c480b69-5456799f87-8s6pm" Jul 08 19:32:55.560754 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:55.560731 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/058f0a1a-31c0-4f1f-8080-0f80df5a9419-maas-api-tls\") pod \"maas-api-e2e-mt-7c480b69-5456799f87-8s6pm\" (UID: \"058f0a1a-31c0-4f1f-8080-0f80df5a9419\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c480b69-5456799f87-8s6pm" Jul 08 19:32:55.717601 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:55.717547 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c480b69-5456799f87-8s6pm" Jul 08 19:32:55.847660 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:55.847634 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7c480b69-5456799f87-8s6pm"] Jul 08 19:32:55.849644 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:32:55.849615 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod058f0a1a_31c0_4f1f_8080_0f80df5a9419.slice/crio-0fb7c0ab824746cc393bbac7bad9f1f846619a2428baab945b4221a99d783288 WatchSource:0}: Error finding container 0fb7c0ab824746cc393bbac7bad9f1f846619a2428baab945b4221a99d783288: Status 404 returned error can't find the container with id 0fb7c0ab824746cc393bbac7bad9f1f846619a2428baab945b4221a99d783288 Jul 08 19:32:55.944991 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:55.944958 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c480b69-5456799f87-8s6pm" event={"ID":"058f0a1a-31c0-4f1f-8080-0f80df5a9419","Type":"ContainerStarted","Data":"0fb7c0ab824746cc393bbac7bad9f1f846619a2428baab945b4221a99d783288"} Jul 08 19:32:56.952389 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:56.952346 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c480b69-5456799f87-8s6pm" event={"ID":"058f0a1a-31c0-4f1f-8080-0f80df5a9419","Type":"ContainerStarted","Data":"c05d0f8502ad3f3bf0f9ff914e7409859827f30b131f58cb9f72a6846979c246"} Jul 08 19:32:56.952874 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:56.952519 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c480b69-5456799f87-8s6pm" Jul 08 19:32:56.970858 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:56.970806 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c480b69-5456799f87-8s6pm" podStartSLOduration=3.557733537 podStartE2EDuration="3.970790246s" totalImagesPullingTime="413.056709ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:32:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:32:56.969259914 +0000 UTC m=+1760.228212841" watchObservedRunningTime="2026-07-08 19:32:56.970790246 +0000 UTC m=+1760.229743170" Jul 08 19:32:58.291879 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:32:58.291845 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:33:02.964033 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:33:02.964001 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c480b69-5456799f87-8s6pm" Jul 08 19:33:14.632374 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:33:14.632334 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:33:35.781880 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:33:35.781767 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:33:36.106412 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:33:36.106328 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7c480b69-5456799f87-8s6pm"] Jul 08 19:33:36.106570 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:33:36.106547 2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c480b69-5456799f87-8s6pm" podUID="058f0a1a-31c0-4f1f-8080-0f80df5a9419" containerName="maas-api" containerID="cri-o://c05d0f8502ad3f3bf0f9ff914e7409859827f30b131f58cb9f72a6846979c246" gracePeriod=30 Jul 08 19:33:36.352611 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:33:36.352562 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c480b69-5456799f87-8s6pm" Jul 08 19:33:36.406365 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:33:36.406337 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:33:36.519046 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:33:36.519012 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbk8z\" (UniqueName: \"kubernetes.io/projected/058f0a1a-31c0-4f1f-8080-0f80df5a9419-kube-api-access-zbk8z\") pod \"058f0a1a-31c0-4f1f-8080-0f80df5a9419\" (UID: \"058f0a1a-31c0-4f1f-8080-0f80df5a9419\") " Jul 08 19:33:36.519204 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:33:36.519063 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/058f0a1a-31c0-4f1f-8080-0f80df5a9419-maas-api-tls\") pod \"058f0a1a-31c0-4f1f-8080-0f80df5a9419\" (UID: \"058f0a1a-31c0-4f1f-8080-0f80df5a9419\") " Jul 08 19:33:36.519204 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:33:36.519095 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/058f0a1a-31c0-4f1f-8080-0f80df5a9419-openshift-service-ca\") pod \"058f0a1a-31c0-4f1f-8080-0f80df5a9419\" (UID: \"058f0a1a-31c0-4f1f-8080-0f80df5a9419\") " Jul 08 19:33:36.519565 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:33:36.519520 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/058f0a1a-31c0-4f1f-8080-0f80df5a9419-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "058f0a1a-31c0-4f1f-8080-0f80df5a9419" (UID: "058f0a1a-31c0-4f1f-8080-0f80df5a9419"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:33:36.521229 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:33:36.521204 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/058f0a1a-31c0-4f1f-8080-0f80df5a9419-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "058f0a1a-31c0-4f1f-8080-0f80df5a9419" (UID: "058f0a1a-31c0-4f1f-8080-0f80df5a9419"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:33:36.521346 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:33:36.521238 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/058f0a1a-31c0-4f1f-8080-0f80df5a9419-kube-api-access-zbk8z" (OuterVolumeSpecName: "kube-api-access-zbk8z") pod "058f0a1a-31c0-4f1f-8080-0f80df5a9419" (UID: "058f0a1a-31c0-4f1f-8080-0f80df5a9419"). InnerVolumeSpecName "kube-api-access-zbk8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:33:36.620614 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:33:36.620565 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbk8z\" (UniqueName: \"kubernetes.io/projected/058f0a1a-31c0-4f1f-8080-0f80df5a9419-kube-api-access-zbk8z\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:33:36.620614 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:33:36.620608 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/058f0a1a-31c0-4f1f-8080-0f80df5a9419-maas-api-tls\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:33:36.620614 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:33:36.620619 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/058f0a1a-31c0-4f1f-8080-0f80df5a9419-openshift-service-ca\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:33:37.158899 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:33:37.158869 2558 generic.go:356] "Generic (PLEG): container finished" podID="058f0a1a-31c0-4f1f-8080-0f80df5a9419" containerID="c05d0f8502ad3f3bf0f9ff914e7409859827f30b131f58cb9f72a6846979c246" exitCode=0 Jul 08 19:33:37.159289 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:33:37.158916 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c480b69-5456799f87-8s6pm" Jul 08 19:33:37.159289 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:33:37.158954 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c480b69-5456799f87-8s6pm" event={"ID":"058f0a1a-31c0-4f1f-8080-0f80df5a9419","Type":"ContainerDied","Data":"c05d0f8502ad3f3bf0f9ff914e7409859827f30b131f58cb9f72a6846979c246"} Jul 08 19:33:37.159289 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:33:37.159009 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c480b69-5456799f87-8s6pm" event={"ID":"058f0a1a-31c0-4f1f-8080-0f80df5a9419","Type":"ContainerDied","Data":"0fb7c0ab824746cc393bbac7bad9f1f846619a2428baab945b4221a99d783288"} Jul 08 19:33:37.159289 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:33:37.159031 2558 scope.go:117] "RemoveContainer" containerID="c05d0f8502ad3f3bf0f9ff914e7409859827f30b131f58cb9f72a6846979c246" Jul 08 19:33:37.168081 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:33:37.168062 2558 scope.go:117] "RemoveContainer" containerID="c05d0f8502ad3f3bf0f9ff914e7409859827f30b131f58cb9f72a6846979c246" Jul 08 19:33:37.168334 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:33:37.168318 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c05d0f8502ad3f3bf0f9ff914e7409859827f30b131f58cb9f72a6846979c246\": container with ID starting with c05d0f8502ad3f3bf0f9ff914e7409859827f30b131f58cb9f72a6846979c246 not found: ID does not exist" containerID="c05d0f8502ad3f3bf0f9ff914e7409859827f30b131f58cb9f72a6846979c246" Jul 08 19:33:37.168382 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:33:37.168340 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c05d0f8502ad3f3bf0f9ff914e7409859827f30b131f58cb9f72a6846979c246"} err="failed to get container status \"c05d0f8502ad3f3bf0f9ff914e7409859827f30b131f58cb9f72a6846979c246\": rpc error: code = NotFound desc = could not find container \"c05d0f8502ad3f3bf0f9ff914e7409859827f30b131f58cb9f72a6846979c246\": container with ID starting with c05d0f8502ad3f3bf0f9ff914e7409859827f30b131f58cb9f72a6846979c246 not found: ID does not exist" Jul 08 19:33:37.181017 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:33:37.180997 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7c480b69-5456799f87-8s6pm"] Jul 08 19:33:37.187400 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:33:37.187379 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7c480b69-5456799f87-8s6pm"] Jul 08 19:33:37.281391 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:33:37.281365 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="058f0a1a-31c0-4f1f-8080-0f80df5a9419" path="/var/lib/kubelet/pods/058f0a1a-31c0-4f1f-8080-0f80df5a9419/volumes" Jul 08 19:33:37.327424 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:33:37.327400 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-54hnr_1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a/console-operator/2.log" Jul 08 19:33:37.328588 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:33:37.328552 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-54hnr_1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a/console-operator/2.log" Jul 08 19:33:41.939564 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:33:41.939527 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:33:46.928036 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:33:46.927995 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:23.946737 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:34:23.946695 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:25.302748 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:34:25.302712 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:29.917835 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:34:29.917799 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:30.003257 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:34:30.003217 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:30.199969 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:34:30.199871 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:31.491530 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:34:31.491480 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:31.584648 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:34:31.584605 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:31.684162 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:34:31.684133 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:32.305023 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:34:32.304984 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:32.479168 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:34:32.479130 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:32.578681 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:34:32.578592 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:32.677980 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:34:32.677939 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:34.056395 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:34:34.056351 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:40.727487 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:34:40.727450 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:40.826135 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:34:40.826103 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:41.421798 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:34:41.421763 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:47.497669 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:34:47.497627 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:47.561762 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:34:47.561730 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:47.648146 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:34:47.648112 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:47.745891 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:34:47.745858 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:58.221065 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:34:58.221028 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:34:58.618088 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:34:58.617998 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:35:19.648474 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:19.648428 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:35:25.573905 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:25.573867 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:35:25.673588 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:25.673547 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:35:25.966763 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:25.966730 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:35:26.285432 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:26.285329 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:35:26.502816 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:26.502780 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:35:29.542114 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:29.542057 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:35:29.625784 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:29.625743 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:35:29.627390 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:29.627363 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8ab38c-549f6844cf-vmjl6"] Jul 08 19:35:29.627794 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:29.627778 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="058f0a1a-31c0-4f1f-8080-0f80df5a9419" containerName="maas-api" Jul 08 19:35:29.627886 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:29.627796 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="058f0a1a-31c0-4f1f-8080-0f80df5a9419" containerName="maas-api" Jul 08 19:35:29.627886 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:29.627863 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="058f0a1a-31c0-4f1f-8080-0f80df5a9419" containerName="maas-api" Jul 08 19:35:29.630879 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:29.630859 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ab38c-549f6844cf-vmjl6" Jul 08 19:35:29.633924 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:29.633904 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-8ab38c-serving-cert\"" Jul 08 19:35:29.641148 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:29.641126 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8ab38c-549f6844cf-vmjl6"] Jul 08 19:35:29.677872 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:29.677839 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkjbm\" (UniqueName: \"kubernetes.io/projected/a96a0099-e387-47af-ac4d-549a7028aa3f-kube-api-access-vkjbm\") pod \"maas-api-e2e-inf-a-8ab38c-549f6844cf-vmjl6\" (UID: \"a96a0099-e387-47af-ac4d-549a7028aa3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ab38c-549f6844cf-vmjl6" Jul 08 19:35:29.678032 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:29.677885 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a96a0099-e387-47af-ac4d-549a7028aa3f-maas-api-tls\") pod \"maas-api-e2e-inf-a-8ab38c-549f6844cf-vmjl6\" (UID: \"a96a0099-e387-47af-ac4d-549a7028aa3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ab38c-549f6844cf-vmjl6" Jul 08 19:35:29.678032 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:29.677934 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a96a0099-e387-47af-ac4d-549a7028aa3f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8ab38c-549f6844cf-vmjl6\" (UID: \"a96a0099-e387-47af-ac4d-549a7028aa3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ab38c-549f6844cf-vmjl6" Jul 08 19:35:29.696590 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:29.696554 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:35:29.779345 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:29.779309 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkjbm\" (UniqueName: \"kubernetes.io/projected/a96a0099-e387-47af-ac4d-549a7028aa3f-kube-api-access-vkjbm\") pod \"maas-api-e2e-inf-a-8ab38c-549f6844cf-vmjl6\" (UID: \"a96a0099-e387-47af-ac4d-549a7028aa3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ab38c-549f6844cf-vmjl6" Jul 08 19:35:29.779512 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:29.779361 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a96a0099-e387-47af-ac4d-549a7028aa3f-maas-api-tls\") pod \"maas-api-e2e-inf-a-8ab38c-549f6844cf-vmjl6\" (UID: \"a96a0099-e387-47af-ac4d-549a7028aa3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ab38c-549f6844cf-vmjl6" Jul 08 19:35:29.779512 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:29.779386 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a96a0099-e387-47af-ac4d-549a7028aa3f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8ab38c-549f6844cf-vmjl6\" (UID: \"a96a0099-e387-47af-ac4d-549a7028aa3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ab38c-549f6844cf-vmjl6" Jul 08 19:35:29.779659 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:35:29.779528 2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-8ab38c-serving-cert: secret "maas-api-e2e-inf-a-8ab38c-serving-cert" not found Jul 08 19:35:29.779659 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:35:29.779649 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a96a0099-e387-47af-ac4d-549a7028aa3f-maas-api-tls podName:a96a0099-e387-47af-ac4d-549a7028aa3f nodeName:}" failed. No retries permitted until 2026-07-08 19:35:30.279631757 +0000 UTC m=+1913.538584660 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a96a0099-e387-47af-ac4d-549a7028aa3f-maas-api-tls") pod "maas-api-e2e-inf-a-8ab38c-549f6844cf-vmjl6" (UID: "a96a0099-e387-47af-ac4d-549a7028aa3f") : secret "maas-api-e2e-inf-a-8ab38c-serving-cert" not found Jul 08 19:35:29.780009 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:29.779990 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a96a0099-e387-47af-ac4d-549a7028aa3f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8ab38c-549f6844cf-vmjl6\" (UID: \"a96a0099-e387-47af-ac4d-549a7028aa3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ab38c-549f6844cf-vmjl6" Jul 08 19:35:29.788734 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:29.788711 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkjbm\" (UniqueName: \"kubernetes.io/projected/a96a0099-e387-47af-ac4d-549a7028aa3f-kube-api-access-vkjbm\") pod \"maas-api-e2e-inf-a-8ab38c-549f6844cf-vmjl6\" (UID: \"a96a0099-e387-47af-ac4d-549a7028aa3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ab38c-549f6844cf-vmjl6" Jul 08 19:35:29.966843 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:29.966810 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:35:30.112955 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:30.112920 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:35:30.214338 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:30.214301 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:35:30.283632 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:30.283532 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a96a0099-e387-47af-ac4d-549a7028aa3f-maas-api-tls\") pod \"maas-api-e2e-inf-a-8ab38c-549f6844cf-vmjl6\" (UID: \"a96a0099-e387-47af-ac4d-549a7028aa3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ab38c-549f6844cf-vmjl6" Jul 08 19:35:30.286025 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:30.285988 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a96a0099-e387-47af-ac4d-549a7028aa3f-maas-api-tls\") pod \"maas-api-e2e-inf-a-8ab38c-549f6844cf-vmjl6\" (UID: \"a96a0099-e387-47af-ac4d-549a7028aa3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ab38c-549f6844cf-vmjl6" Jul 08 19:35:30.542378 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:30.542297 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ab38c-549f6844cf-vmjl6" Jul 08 19:35:30.679029 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:30.679001 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8ab38c-549f6844cf-vmjl6"] Jul 08 19:35:30.680513 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:35:30.680486 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda96a0099_e387_47af_ac4d_549a7028aa3f.slice/crio-b5e6c11402f778a418d6c2c578571ccc8bdb7f5f83c9cb89ea07bfb5f51fa924 WatchSource:0}: Error finding container b5e6c11402f778a418d6c2c578571ccc8bdb7f5f83c9cb89ea07bfb5f51fa924: Status 404 returned error can't find the container with id b5e6c11402f778a418d6c2c578571ccc8bdb7f5f83c9cb89ea07bfb5f51fa924 Jul 08 19:35:30.681989 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:30.681958 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 19:35:30.732973 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:30.732942 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ab38c-549f6844cf-vmjl6" event={"ID":"a96a0099-e387-47af-ac4d-549a7028aa3f","Type":"ContainerStarted","Data":"b5e6c11402f778a418d6c2c578571ccc8bdb7f5f83c9cb89ea07bfb5f51fa924"} Jul 08 19:35:31.739849 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:31.739814 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ab38c-549f6844cf-vmjl6" event={"ID":"a96a0099-e387-47af-ac4d-549a7028aa3f","Type":"ContainerStarted","Data":"1ec0ad0048e466dc2a653411e6cc7005d20810a640e2a33e6f8999ea49d57140"} Jul 08 19:35:31.740219 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:31.739936 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ab38c-549f6844cf-vmjl6" Jul 08 19:35:31.762192 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:31.762144 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ab38c-549f6844cf-vmjl6" podStartSLOduration=2.2509607750000002 podStartE2EDuration="2.762129595s" totalImagesPullingTime="511.16882ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:35:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:35:31.760820903 +0000 UTC m=+1915.019773829" watchObservedRunningTime="2026-07-08 19:35:31.762129595 +0000 UTC m=+1915.021082519" Jul 08 19:35:34.127434 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:34.127395 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:35:34.513199 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:34.513104 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:35:34.817818 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:34.817721 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:35:35.290855 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:35.290817 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:35:37.665747 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:37.665707 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:35:37.750006 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:37.749979 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ab38c-549f6844cf-vmjl6" Jul 08 19:35:37.777080 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:37.777046 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:35:37.791067 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:37.791034 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1709ef-985f456f-rjzw7"] Jul 08 19:35:37.795153 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:37.795132 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1709ef-985f456f-rjzw7" Jul 08 19:35:37.798104 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:37.798085 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-1709ef-serving-cert\"" Jul 08 19:35:37.810277 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:37.810256 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1709ef-985f456f-rjzw7"] Jul 08 19:35:37.845903 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:37.845879 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6crc\" (UniqueName: \"kubernetes.io/projected/9949f580-d74b-4bf5-beed-4ba51ef6d8bf-kube-api-access-m6crc\") pod \"maas-api-e2e-inf-b-1709ef-985f456f-rjzw7\" (UID: \"9949f580-d74b-4bf5-beed-4ba51ef6d8bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1709ef-985f456f-rjzw7" Jul 08 19:35:37.846041 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:37.845960 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9949f580-d74b-4bf5-beed-4ba51ef6d8bf-maas-api-tls\") pod \"maas-api-e2e-inf-b-1709ef-985f456f-rjzw7\" (UID: \"9949f580-d74b-4bf5-beed-4ba51ef6d8bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1709ef-985f456f-rjzw7" Jul 08 19:35:37.846041 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:37.845978 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9949f580-d74b-4bf5-beed-4ba51ef6d8bf-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1709ef-985f456f-rjzw7\" (UID: \"9949f580-d74b-4bf5-beed-4ba51ef6d8bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1709ef-985f456f-rjzw7" Jul 08 19:35:37.877500 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:37.877473 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:35:37.947500 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:37.947425 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6crc\" (UniqueName: \"kubernetes.io/projected/9949f580-d74b-4bf5-beed-4ba51ef6d8bf-kube-api-access-m6crc\") pod \"maas-api-e2e-inf-b-1709ef-985f456f-rjzw7\" (UID: \"9949f580-d74b-4bf5-beed-4ba51ef6d8bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1709ef-985f456f-rjzw7" Jul 08 19:35:37.947500 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:37.947485 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9949f580-d74b-4bf5-beed-4ba51ef6d8bf-maas-api-tls\") pod \"maas-api-e2e-inf-b-1709ef-985f456f-rjzw7\" (UID: \"9949f580-d74b-4bf5-beed-4ba51ef6d8bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1709ef-985f456f-rjzw7" Jul 08 19:35:37.947742 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:37.947518 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9949f580-d74b-4bf5-beed-4ba51ef6d8bf-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1709ef-985f456f-rjzw7\" (UID: \"9949f580-d74b-4bf5-beed-4ba51ef6d8bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1709ef-985f456f-rjzw7" Jul 08 19:35:37.947742 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:35:37.947673 2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-1709ef-serving-cert: secret "maas-api-e2e-inf-b-1709ef-serving-cert" not found Jul 08 19:35:37.947849 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:35:37.947773 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9949f580-d74b-4bf5-beed-4ba51ef6d8bf-maas-api-tls podName:9949f580-d74b-4bf5-beed-4ba51ef6d8bf nodeName:}" failed. No retries permitted until 2026-07-08 19:35:38.447740369 +0000 UTC m=+1921.706693274 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9949f580-d74b-4bf5-beed-4ba51ef6d8bf-maas-api-tls") pod "maas-api-e2e-inf-b-1709ef-985f456f-rjzw7" (UID: "9949f580-d74b-4bf5-beed-4ba51ef6d8bf") : secret "maas-api-e2e-inf-b-1709ef-serving-cert" not found Jul 08 19:35:37.948229 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:37.948210 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9949f580-d74b-4bf5-beed-4ba51ef6d8bf-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1709ef-985f456f-rjzw7\" (UID: \"9949f580-d74b-4bf5-beed-4ba51ef6d8bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1709ef-985f456f-rjzw7" Jul 08 19:35:37.963795 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:37.963772 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6crc\" (UniqueName: \"kubernetes.io/projected/9949f580-d74b-4bf5-beed-4ba51ef6d8bf-kube-api-access-m6crc\") pod \"maas-api-e2e-inf-b-1709ef-985f456f-rjzw7\" (UID: \"9949f580-d74b-4bf5-beed-4ba51ef6d8bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1709ef-985f456f-rjzw7" Jul 08 19:35:38.089936 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:38.089882 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:35:38.451039 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:38.451009 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9949f580-d74b-4bf5-beed-4ba51ef6d8bf-maas-api-tls\") pod \"maas-api-e2e-inf-b-1709ef-985f456f-rjzw7\" (UID: \"9949f580-d74b-4bf5-beed-4ba51ef6d8bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1709ef-985f456f-rjzw7" Jul 08 19:35:38.453170 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:38.453148 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9949f580-d74b-4bf5-beed-4ba51ef6d8bf-maas-api-tls\") pod \"maas-api-e2e-inf-b-1709ef-985f456f-rjzw7\" (UID: \"9949f580-d74b-4bf5-beed-4ba51ef6d8bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1709ef-985f456f-rjzw7" Jul 08 19:35:38.466388 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:38.466362 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:35:38.564476 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:38.564442 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:35:38.706291 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:38.706208 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1709ef-985f456f-rjzw7" Jul 08 19:35:38.849767 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:38.849738 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1709ef-985f456f-rjzw7"] Jul 08 19:35:38.851333 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:35:38.851310 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9949f580_d74b_4bf5_beed_4ba51ef6d8bf.slice/crio-71f8673e2c533575cfdd79228d0941f5021dd926d877fe4a571960d488efd2fb WatchSource:0}: Error finding container 71f8673e2c533575cfdd79228d0941f5021dd926d877fe4a571960d488efd2fb: Status 404 returned error can't find the container with id 71f8673e2c533575cfdd79228d0941f5021dd926d877fe4a571960d488efd2fb Jul 08 19:35:39.787292 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:39.787195 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1709ef-985f456f-rjzw7" event={"ID":"9949f580-d74b-4bf5-beed-4ba51ef6d8bf","Type":"ContainerStarted","Data":"e69bc38efc1e2782e2549f39cf6af865db4f936291af128eeb2a8c80d8e98f7a"} Jul 08 19:35:39.787771 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:39.787293 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1709ef-985f456f-rjzw7" event={"ID":"9949f580-d74b-4bf5-beed-4ba51ef6d8bf","Type":"ContainerStarted","Data":"71f8673e2c533575cfdd79228d0941f5021dd926d877fe4a571960d488efd2fb"} Jul 08 19:35:39.787771 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:39.787364 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1709ef-985f456f-rjzw7" Jul 08 19:35:39.832773 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:39.832727 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1709ef-985f456f-rjzw7" podStartSLOduration=2.279023037 podStartE2EDuration="2.832710894s" totalImagesPullingTime="553.687857ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:35:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:35:39.822479207 +0000 UTC m=+1923.081432132" watchObservedRunningTime="2026-07-08 19:35:39.832710894 +0000 UTC m=+1923.091663819" Jul 08 19:35:42.121172 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:42.121137 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:35:45.798780 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:45.798748 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1709ef-985f456f-rjzw7" Jul 08 19:35:52.250364 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:52.250321 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:35:52.350963 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:52.350927 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:35:58.129510 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:58.129471 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:35:59.333901 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:59.333811 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:35:59.432157 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:35:59.432116 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:36:28.113357 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:36:28.113319 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:36:41.485178 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:36:41.485139 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:36:41.594654 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:36:41.594616 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:36:56.253812 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:36:56.253769 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:36:57.218397 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:36:57.218356 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:36:57.362622 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:36:57.362564 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:36:57.769694 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:36:57.769662 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:36:57.868402 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:36:57.868363 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:36:57.970763 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:36:57.970716 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:36:58.070892 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:36:58.070805 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8ab38c-549f6844cf-vmjl6"] Jul 08 19:36:58.071091 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:36:58.071066 2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ab38c-549f6844cf-vmjl6" podUID="a96a0099-e387-47af-ac4d-549a7028aa3f" containerName="maas-api" containerID="cri-o://1ec0ad0048e466dc2a653411e6cc7005d20810a640e2a33e6f8999ea49d57140" gracePeriod=30 Jul 08 19:36:58.097191 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:36:58.097163 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:36:58.191635 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:36:58.191607 2558 generic.go:356] "Generic (PLEG): container finished" podID="a96a0099-e387-47af-ac4d-549a7028aa3f" containerID="1ec0ad0048e466dc2a653411e6cc7005d20810a640e2a33e6f8999ea49d57140" exitCode=0 Jul 08 19:36:58.191778 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:36:58.191652 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ab38c-549f6844cf-vmjl6" event={"ID":"a96a0099-e387-47af-ac4d-549a7028aa3f","Type":"ContainerDied","Data":"1ec0ad0048e466dc2a653411e6cc7005d20810a640e2a33e6f8999ea49d57140"} Jul 08 19:36:58.321609 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:36:58.321534 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ab38c-549f6844cf-vmjl6" Jul 08 19:36:58.450378 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:36:58.450351 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a96a0099-e387-47af-ac4d-549a7028aa3f-maas-api-tls\") pod \"a96a0099-e387-47af-ac4d-549a7028aa3f\" (UID: \"a96a0099-e387-47af-ac4d-549a7028aa3f\") " Jul 08 19:36:58.450827 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:36:58.450406 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkjbm\" (UniqueName: \"kubernetes.io/projected/a96a0099-e387-47af-ac4d-549a7028aa3f-kube-api-access-vkjbm\") pod \"a96a0099-e387-47af-ac4d-549a7028aa3f\" (UID: \"a96a0099-e387-47af-ac4d-549a7028aa3f\") " Jul 08 19:36:58.450827 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:36:58.450476 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a96a0099-e387-47af-ac4d-549a7028aa3f-openshift-service-ca\") pod \"a96a0099-e387-47af-ac4d-549a7028aa3f\" (UID: \"a96a0099-e387-47af-ac4d-549a7028aa3f\") " Jul 08 19:36:58.450907 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:36:58.450851 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a96a0099-e387-47af-ac4d-549a7028aa3f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a96a0099-e387-47af-ac4d-549a7028aa3f" (UID: "a96a0099-e387-47af-ac4d-549a7028aa3f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:36:58.452358 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:36:58.452332 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a96a0099-e387-47af-ac4d-549a7028aa3f-kube-api-access-vkjbm" (OuterVolumeSpecName: "kube-api-access-vkjbm") pod "a96a0099-e387-47af-ac4d-549a7028aa3f" (UID: "a96a0099-e387-47af-ac4d-549a7028aa3f"). InnerVolumeSpecName "kube-api-access-vkjbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:36:58.452450 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:36:58.452413 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a96a0099-e387-47af-ac4d-549a7028aa3f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a96a0099-e387-47af-ac4d-549a7028aa3f" (UID: "a96a0099-e387-47af-ac4d-549a7028aa3f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:36:58.552052 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:36:58.552021 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a96a0099-e387-47af-ac4d-549a7028aa3f-maas-api-tls\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:36:58.552052 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:36:58.552053 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkjbm\" (UniqueName: \"kubernetes.io/projected/a96a0099-e387-47af-ac4d-549a7028aa3f-kube-api-access-vkjbm\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:36:58.552052 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:36:58.552070 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a96a0099-e387-47af-ac4d-549a7028aa3f-openshift-service-ca\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:36:59.199188 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:36:59.199156 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ab38c-549f6844cf-vmjl6" Jul 08 19:36:59.199409 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:36:59.199194 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ab38c-549f6844cf-vmjl6" event={"ID":"a96a0099-e387-47af-ac4d-549a7028aa3f","Type":"ContainerDied","Data":"b5e6c11402f778a418d6c2c578571ccc8bdb7f5f83c9cb89ea07bfb5f51fa924"} Jul 08 19:36:59.199409 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:36:59.199262 2558 scope.go:117] "RemoveContainer" containerID="1ec0ad0048e466dc2a653411e6cc7005d20810a640e2a33e6f8999ea49d57140" Jul 08 19:36:59.250560 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:36:59.250530 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8ab38c-549f6844cf-vmjl6"] Jul 08 19:36:59.260691 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:36:59.260666 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8ab38c-549f6844cf-vmjl6"] Jul 08 19:36:59.282139 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:36:59.282062 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a96a0099-e387-47af-ac4d-549a7028aa3f" path="/var/lib/kubelet/pods/a96a0099-e387-47af-ac4d-549a7028aa3f/volumes" Jul 08 19:37:02.862464 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:02.862421 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:37:40.055924 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:40.055889 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:37:40.343880 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:40.343796 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:37:40.754553 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:40.754504 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:37:41.170600 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:41.170536 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:37:41.271007 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:41.270976 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:37:41.371557 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:41.371496 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:37:41.593617 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:41.593526 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1709ef-985f456f-rjzw7"] Jul 08 19:37:41.593814 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:41.593784 2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1709ef-985f456f-rjzw7" podUID="9949f580-d74b-4bf5-beed-4ba51ef6d8bf" containerName="maas-api" containerID="cri-o://e69bc38efc1e2782e2549f39cf6af865db4f936291af128eeb2a8c80d8e98f7a" gracePeriod=30 Jul 08 19:37:41.629064 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:41.629026 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:37:41.842070 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:41.842046 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1709ef-985f456f-rjzw7" Jul 08 19:37:41.924318 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:41.924290 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6crc\" (UniqueName: \"kubernetes.io/projected/9949f580-d74b-4bf5-beed-4ba51ef6d8bf-kube-api-access-m6crc\") pod \"9949f580-d74b-4bf5-beed-4ba51ef6d8bf\" (UID: \"9949f580-d74b-4bf5-beed-4ba51ef6d8bf\") " Jul 08 19:37:41.924498 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:41.924348 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9949f580-d74b-4bf5-beed-4ba51ef6d8bf-openshift-service-ca\") pod \"9949f580-d74b-4bf5-beed-4ba51ef6d8bf\" (UID: \"9949f580-d74b-4bf5-beed-4ba51ef6d8bf\") " Jul 08 19:37:41.924498 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:41.924464 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9949f580-d74b-4bf5-beed-4ba51ef6d8bf-maas-api-tls\") pod \"9949f580-d74b-4bf5-beed-4ba51ef6d8bf\" (UID: \"9949f580-d74b-4bf5-beed-4ba51ef6d8bf\") " Jul 08 19:37:41.924698 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:41.924678 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9949f580-d74b-4bf5-beed-4ba51ef6d8bf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9949f580-d74b-4bf5-beed-4ba51ef6d8bf" (UID: "9949f580-d74b-4bf5-beed-4ba51ef6d8bf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:37:41.924818 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:41.924805 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9949f580-d74b-4bf5-beed-4ba51ef6d8bf-openshift-service-ca\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:37:41.926356 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:41.926332 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9949f580-d74b-4bf5-beed-4ba51ef6d8bf-kube-api-access-m6crc" (OuterVolumeSpecName: "kube-api-access-m6crc") pod "9949f580-d74b-4bf5-beed-4ba51ef6d8bf" (UID: "9949f580-d74b-4bf5-beed-4ba51ef6d8bf"). InnerVolumeSpecName "kube-api-access-m6crc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:37:41.926451 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:41.926409 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9949f580-d74b-4bf5-beed-4ba51ef6d8bf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9949f580-d74b-4bf5-beed-4ba51ef6d8bf" (UID: "9949f580-d74b-4bf5-beed-4ba51ef6d8bf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:37:42.025903 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:42.025864 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9949f580-d74b-4bf5-beed-4ba51ef6d8bf-maas-api-tls\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:37:42.025903 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:42.025897 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6crc\" (UniqueName: \"kubernetes.io/projected/9949f580-d74b-4bf5-beed-4ba51ef6d8bf-kube-api-access-m6crc\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:37:42.417794 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:42.417757 2558 generic.go:356] "Generic (PLEG): container finished" podID="9949f580-d74b-4bf5-beed-4ba51ef6d8bf" containerID="e69bc38efc1e2782e2549f39cf6af865db4f936291af128eeb2a8c80d8e98f7a" exitCode=0 Jul 08 19:37:42.418248 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:42.417816 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1709ef-985f456f-rjzw7" Jul 08 19:37:42.418248 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:42.417842 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1709ef-985f456f-rjzw7" event={"ID":"9949f580-d74b-4bf5-beed-4ba51ef6d8bf","Type":"ContainerDied","Data":"e69bc38efc1e2782e2549f39cf6af865db4f936291af128eeb2a8c80d8e98f7a"} Jul 08 19:37:42.418248 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:42.417922 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1709ef-985f456f-rjzw7" event={"ID":"9949f580-d74b-4bf5-beed-4ba51ef6d8bf","Type":"ContainerDied","Data":"71f8673e2c533575cfdd79228d0941f5021dd926d877fe4a571960d488efd2fb"} Jul 08 19:37:42.418248 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:42.417946 2558 scope.go:117] "RemoveContainer" containerID="e69bc38efc1e2782e2549f39cf6af865db4f936291af128eeb2a8c80d8e98f7a" Jul 08 19:37:42.431682 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:42.431663 2558 scope.go:117] "RemoveContainer" containerID="e69bc38efc1e2782e2549f39cf6af865db4f936291af128eeb2a8c80d8e98f7a" Jul 08 19:37:42.431934 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:37:42.431913 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e69bc38efc1e2782e2549f39cf6af865db4f936291af128eeb2a8c80d8e98f7a\": container with ID starting with e69bc38efc1e2782e2549f39cf6af865db4f936291af128eeb2a8c80d8e98f7a not found: ID does not exist" containerID="e69bc38efc1e2782e2549f39cf6af865db4f936291af128eeb2a8c80d8e98f7a" Jul 08 19:37:42.432021 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:42.431945 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e69bc38efc1e2782e2549f39cf6af865db4f936291af128eeb2a8c80d8e98f7a"} err="failed to get container status \"e69bc38efc1e2782e2549f39cf6af865db4f936291af128eeb2a8c80d8e98f7a\": rpc error: code = NotFound desc = could not find container \"e69bc38efc1e2782e2549f39cf6af865db4f936291af128eeb2a8c80d8e98f7a\": container with ID starting with e69bc38efc1e2782e2549f39cf6af865db4f936291af128eeb2a8c80d8e98f7a not found: ID does not exist" Jul 08 19:37:42.441161 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:42.441141 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1709ef-985f456f-rjzw7"] Jul 08 19:37:42.448008 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:42.447990 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1709ef-985f456f-rjzw7"] Jul 08 19:37:43.281992 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:43.281959 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9949f580-d74b-4bf5-beed-4ba51ef6d8bf" path="/var/lib/kubelet/pods/9949f580-d74b-4bf5-beed-4ba51ef6d8bf/volumes" Jul 08 19:37:47.422023 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:47.421987 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:37:47.529735 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:47.529698 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:37:47.712331 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:47.712242 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:37:47.856718 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:47.856657 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:37:47.957698 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:47.957650 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:37:48.289022 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:48.288989 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:37:48.608461 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:48.608362 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:37:48.965093 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:48.965060 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:37:51.255146 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:51.255112 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:37:51.347949 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:51.347914 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:37:51.401070 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:51.401024 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5f8b33-754f4c77db-pw97v"] Jul 08 19:37:51.401384 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:51.401372 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a96a0099-e387-47af-ac4d-549a7028aa3f" containerName="maas-api" Jul 08 19:37:51.401428 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:51.401385 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a96a0099-e387-47af-ac4d-549a7028aa3f" containerName="maas-api" Jul 08 19:37:51.401428 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:51.401402 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9949f580-d74b-4bf5-beed-4ba51ef6d8bf" containerName="maas-api" Jul 08 19:37:51.401428 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:51.401408 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9949f580-d74b-4bf5-beed-4ba51ef6d8bf" containerName="maas-api" Jul 08 19:37:51.401520 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:51.401467 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a96a0099-e387-47af-ac4d-549a7028aa3f" containerName="maas-api" Jul 08 19:37:51.401520 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:51.401475 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="9949f580-d74b-4bf5-beed-4ba51ef6d8bf" containerName="maas-api" Jul 08 19:37:51.405904 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:51.405888 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5f8b33-754f4c77db-pw97v" Jul 08 19:37:51.408780 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:51.408761 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-5f8b33-serving-cert\"" Jul 08 19:37:51.423184 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:51.423158 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5f8b33-754f4c77db-pw97v"] Jul 08 19:37:51.430002 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:51.429978 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:37:51.511674 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:51.511565 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/101eb7b8-0e9e-4798-886c-2d247a538bed-openshift-service-ca\") pod \"maas-api-e2e-api-a-5f8b33-754f4c77db-pw97v\" (UID: \"101eb7b8-0e9e-4798-886c-2d247a538bed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5f8b33-754f4c77db-pw97v" Jul 08 19:37:51.511827 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:51.511696 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrkxd\" (UniqueName: \"kubernetes.io/projected/101eb7b8-0e9e-4798-886c-2d247a538bed-kube-api-access-nrkxd\") pod \"maas-api-e2e-api-a-5f8b33-754f4c77db-pw97v\" (UID: \"101eb7b8-0e9e-4798-886c-2d247a538bed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5f8b33-754f4c77db-pw97v" Jul 08 19:37:51.511827 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:51.511744 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/101eb7b8-0e9e-4798-886c-2d247a538bed-maas-api-tls\") pod \"maas-api-e2e-api-a-5f8b33-754f4c77db-pw97v\" (UID: \"101eb7b8-0e9e-4798-886c-2d247a538bed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5f8b33-754f4c77db-pw97v" Jul 08 19:37:51.612997 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:51.612965 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrkxd\" (UniqueName: \"kubernetes.io/projected/101eb7b8-0e9e-4798-886c-2d247a538bed-kube-api-access-nrkxd\") pod \"maas-api-e2e-api-a-5f8b33-754f4c77db-pw97v\" (UID: \"101eb7b8-0e9e-4798-886c-2d247a538bed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5f8b33-754f4c77db-pw97v" Jul 08 19:37:51.613158 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:51.613009 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/101eb7b8-0e9e-4798-886c-2d247a538bed-maas-api-tls\") pod \"maas-api-e2e-api-a-5f8b33-754f4c77db-pw97v\" (UID: \"101eb7b8-0e9e-4798-886c-2d247a538bed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5f8b33-754f4c77db-pw97v" Jul 08 19:37:51.613158 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:51.613068 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/101eb7b8-0e9e-4798-886c-2d247a538bed-openshift-service-ca\") pod \"maas-api-e2e-api-a-5f8b33-754f4c77db-pw97v\" (UID: \"101eb7b8-0e9e-4798-886c-2d247a538bed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5f8b33-754f4c77db-pw97v" Jul 08 19:37:51.613251 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:37:51.613182 2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-5f8b33-serving-cert: secret "maas-api-e2e-api-a-5f8b33-serving-cert" not found Jul 08 19:37:51.613251 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:37:51.613247 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/101eb7b8-0e9e-4798-886c-2d247a538bed-maas-api-tls podName:101eb7b8-0e9e-4798-886c-2d247a538bed nodeName:}" failed. No retries permitted until 2026-07-08 19:37:52.113228479 +0000 UTC m=+2055.372181393 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/101eb7b8-0e9e-4798-886c-2d247a538bed-maas-api-tls") pod "maas-api-e2e-api-a-5f8b33-754f4c77db-pw97v" (UID: "101eb7b8-0e9e-4798-886c-2d247a538bed") : secret "maas-api-e2e-api-a-5f8b33-serving-cert" not found Jul 08 19:37:51.613759 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:51.613741 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/101eb7b8-0e9e-4798-886c-2d247a538bed-openshift-service-ca\") pod \"maas-api-e2e-api-a-5f8b33-754f4c77db-pw97v\" (UID: \"101eb7b8-0e9e-4798-886c-2d247a538bed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5f8b33-754f4c77db-pw97v" Jul 08 19:37:51.631123 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:51.631101 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrkxd\" (UniqueName: \"kubernetes.io/projected/101eb7b8-0e9e-4798-886c-2d247a538bed-kube-api-access-nrkxd\") pod \"maas-api-e2e-api-a-5f8b33-754f4c77db-pw97v\" (UID: \"101eb7b8-0e9e-4798-886c-2d247a538bed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5f8b33-754f4c77db-pw97v" Jul 08 19:37:51.683040 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:51.683002 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:37:51.966053 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:51.966019 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:37:52.066568 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:52.066535 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:37:52.116894 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:52.116857 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/101eb7b8-0e9e-4798-886c-2d247a538bed-maas-api-tls\") pod \"maas-api-e2e-api-a-5f8b33-754f4c77db-pw97v\" (UID: \"101eb7b8-0e9e-4798-886c-2d247a538bed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5f8b33-754f4c77db-pw97v" Jul 08 19:37:52.119294 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:52.119250 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/101eb7b8-0e9e-4798-886c-2d247a538bed-maas-api-tls\") pod \"maas-api-e2e-api-a-5f8b33-754f4c77db-pw97v\" (UID: \"101eb7b8-0e9e-4798-886c-2d247a538bed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5f8b33-754f4c77db-pw97v" Jul 08 19:37:52.316495 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:52.316387 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5f8b33-754f4c77db-pw97v" Jul 08 19:37:52.458392 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:52.458367 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5f8b33-754f4c77db-pw97v"] Jul 08 19:37:52.460615 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:37:52.460574 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod101eb7b8_0e9e_4798_886c_2d247a538bed.slice/crio-4329fe7c1cc32c3f8fcc524cb176d9c3730c9a4b708aa10840d0109556590458 WatchSource:0}: Error finding container 4329fe7c1cc32c3f8fcc524cb176d9c3730c9a4b708aa10840d0109556590458: Status 404 returned error can't find the container with id 4329fe7c1cc32c3f8fcc524cb176d9c3730c9a4b708aa10840d0109556590458 Jul 08 19:37:52.471629 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:52.471603 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5f8b33-754f4c77db-pw97v" event={"ID":"101eb7b8-0e9e-4798-886c-2d247a538bed","Type":"ContainerStarted","Data":"4329fe7c1cc32c3f8fcc524cb176d9c3730c9a4b708aa10840d0109556590458"} Jul 08 19:37:53.478300 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:53.478267 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5f8b33-754f4c77db-pw97v" event={"ID":"101eb7b8-0e9e-4798-886c-2d247a538bed","Type":"ContainerStarted","Data":"43d776b272fd1968f2b310cb326633d2e234c1e93df4ca5e2598820294dcfdb8"} Jul 08 19:37:53.478694 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:53.478338 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5f8b33-754f4c77db-pw97v" Jul 08 19:37:53.507130 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:53.507081 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5f8b33-754f4c77db-pw97v" podStartSLOduration=2.08023209 podStartE2EDuration="2.507066831s" totalImagesPullingTime="426.834741ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:37:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:37:53.50654214 +0000 UTC m=+2056.765495068" watchObservedRunningTime="2026-07-08 19:37:53.507066831 +0000 UTC m=+2056.766019756" Jul 08 19:37:55.767445 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:55.767404 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:37:56.230312 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:56.230267 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:37:56.552156 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:56.552054 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:37:56.762052 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:56.762002 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:37:59.258533 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:59.258478 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:37:59.348148 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:59.348057 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:37:59.488824 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:59.488756 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5f8b33-754f4c77db-pw97v" Jul 08 19:37:59.512077 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:59.512043 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-39e11a-676f6b9598-vbgb2"] Jul 08 19:37:59.516486 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:59.516459 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e11a-676f6b9598-vbgb2" Jul 08 19:37:59.520811 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:59.520786 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-39e11a-serving-cert\"" Jul 08 19:37:59.529050 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:59.529024 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-39e11a-676f6b9598-vbgb2"] Jul 08 19:37:59.551378 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:59.551346 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:37:59.579752 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:59.579723 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ccd7e0d-b383-4203-9643-f01894ba1810-maas-api-tls\") pod \"maas-api-e2e-api-b-39e11a-676f6b9598-vbgb2\" (UID: \"3ccd7e0d-b383-4203-9643-f01894ba1810\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e11a-676f6b9598-vbgb2" Jul 08 19:37:59.579874 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:59.579778 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm8db\" (UniqueName: \"kubernetes.io/projected/3ccd7e0d-b383-4203-9643-f01894ba1810-kube-api-access-zm8db\") pod \"maas-api-e2e-api-b-39e11a-676f6b9598-vbgb2\" (UID: \"3ccd7e0d-b383-4203-9643-f01894ba1810\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e11a-676f6b9598-vbgb2" Jul 08 19:37:59.579874 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:59.579805 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ccd7e0d-b383-4203-9643-f01894ba1810-openshift-service-ca\") pod \"maas-api-e2e-api-b-39e11a-676f6b9598-vbgb2\" (UID: \"3ccd7e0d-b383-4203-9643-f01894ba1810\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e11a-676f6b9598-vbgb2" Jul 08 19:37:59.680675 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:59.680629 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ccd7e0d-b383-4203-9643-f01894ba1810-openshift-service-ca\") pod \"maas-api-e2e-api-b-39e11a-676f6b9598-vbgb2\" (UID: \"3ccd7e0d-b383-4203-9643-f01894ba1810\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e11a-676f6b9598-vbgb2" Jul 08 19:37:59.680868 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:59.680721 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ccd7e0d-b383-4203-9643-f01894ba1810-maas-api-tls\") pod \"maas-api-e2e-api-b-39e11a-676f6b9598-vbgb2\" (UID: \"3ccd7e0d-b383-4203-9643-f01894ba1810\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e11a-676f6b9598-vbgb2" Jul 08 19:37:59.680868 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:59.680827 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm8db\" (UniqueName: \"kubernetes.io/projected/3ccd7e0d-b383-4203-9643-f01894ba1810-kube-api-access-zm8db\") pod \"maas-api-e2e-api-b-39e11a-676f6b9598-vbgb2\" (UID: \"3ccd7e0d-b383-4203-9643-f01894ba1810\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e11a-676f6b9598-vbgb2" Jul 08 19:37:59.681366 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:59.681339 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ccd7e0d-b383-4203-9643-f01894ba1810-openshift-service-ca\") pod \"maas-api-e2e-api-b-39e11a-676f6b9598-vbgb2\" (UID: \"3ccd7e0d-b383-4203-9643-f01894ba1810\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e11a-676f6b9598-vbgb2" Jul 08 19:37:59.683202 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:59.683178 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ccd7e0d-b383-4203-9643-f01894ba1810-maas-api-tls\") pod \"maas-api-e2e-api-b-39e11a-676f6b9598-vbgb2\" (UID: \"3ccd7e0d-b383-4203-9643-f01894ba1810\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e11a-676f6b9598-vbgb2" Jul 08 19:37:59.689695 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:59.689670 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm8db\" (UniqueName: \"kubernetes.io/projected/3ccd7e0d-b383-4203-9643-f01894ba1810-kube-api-access-zm8db\") pod \"maas-api-e2e-api-b-39e11a-676f6b9598-vbgb2\" (UID: \"3ccd7e0d-b383-4203-9643-f01894ba1810\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e11a-676f6b9598-vbgb2" Jul 08 19:37:59.708676 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:59.708642 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:37:59.829820 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:59.829783 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e11a-676f6b9598-vbgb2" Jul 08 19:37:59.979916 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:37:59.979892 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-39e11a-676f6b9598-vbgb2"] Jul 08 19:37:59.982323 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:37:59.982299 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ccd7e0d_b383_4203_9643_f01894ba1810.slice/crio-4efaeca2876fd61696d82cbd83f01f5c4e3768a037da1743363182b64a7e7146 WatchSource:0}: Error finding container 4efaeca2876fd61696d82cbd83f01f5c4e3768a037da1743363182b64a7e7146: Status 404 returned error can't find the container with id 4efaeca2876fd61696d82cbd83f01f5c4e3768a037da1743363182b64a7e7146 Jul 08 19:38:00.020023 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:00.019990 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:00.124105 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:00.124069 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:00.517644 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:00.517604 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e11a-676f6b9598-vbgb2" event={"ID":"3ccd7e0d-b383-4203-9643-f01894ba1810","Type":"ContainerStarted","Data":"076cb19e7281fad2065ebe7cff2641a960622aaa53fc09d59114b7dbf142389c"} Jul 08 19:38:00.518012 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:00.517758 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e11a-676f6b9598-vbgb2" Jul 08 19:38:00.518012 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:00.517780 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e11a-676f6b9598-vbgb2" event={"ID":"3ccd7e0d-b383-4203-9643-f01894ba1810","Type":"ContainerStarted","Data":"4efaeca2876fd61696d82cbd83f01f5c4e3768a037da1743363182b64a7e7146"} Jul 08 19:38:00.540923 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:00.540879 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e11a-676f6b9598-vbgb2" podStartSLOduration=1.159278466 podStartE2EDuration="1.540862386s" totalImagesPullingTime="381.58392ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:37:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:38:00.537855622 +0000 UTC m=+2063.796808545" watchObservedRunningTime="2026-07-08 19:38:00.540862386 +0000 UTC m=+2063.799815356" Jul 08 19:38:03.747965 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:03.747926 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:06.529056 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:06.529020 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e11a-676f6b9598-vbgb2" Jul 08 19:38:11.905560 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:11.905504 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:12.338381 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:12.338285 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:12.486852 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:12.486815 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:12.585800 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:12.585769 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:12.759733 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:12.759695 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5f8b33-754f4c77db-pw97v"] Jul 08 19:38:12.759996 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:12.759964 2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5f8b33-754f4c77db-pw97v" podUID="101eb7b8-0e9e-4798-886c-2d247a538bed" containerName="maas-api" containerID="cri-o://43d776b272fd1968f2b310cb326633d2e234c1e93df4ca5e2598820294dcfdb8" gracePeriod=30 Jul 08 19:38:12.784467 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:12.784444 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:13.015189 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:13.015112 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5f8b33-754f4c77db-pw97v" Jul 08 19:38:13.092346 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:13.092315 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/101eb7b8-0e9e-4798-886c-2d247a538bed-openshift-service-ca\") pod \"101eb7b8-0e9e-4798-886c-2d247a538bed\" (UID: \"101eb7b8-0e9e-4798-886c-2d247a538bed\") " Jul 08 19:38:13.092510 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:13.092374 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrkxd\" (UniqueName: \"kubernetes.io/projected/101eb7b8-0e9e-4798-886c-2d247a538bed-kube-api-access-nrkxd\") pod \"101eb7b8-0e9e-4798-886c-2d247a538bed\" (UID: \"101eb7b8-0e9e-4798-886c-2d247a538bed\") " Jul 08 19:38:13.092510 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:13.092407 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/101eb7b8-0e9e-4798-886c-2d247a538bed-maas-api-tls\") pod \"101eb7b8-0e9e-4798-886c-2d247a538bed\" (UID: \"101eb7b8-0e9e-4798-886c-2d247a538bed\") " Jul 08 19:38:13.092791 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:13.092758 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/101eb7b8-0e9e-4798-886c-2d247a538bed-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "101eb7b8-0e9e-4798-886c-2d247a538bed" (UID: "101eb7b8-0e9e-4798-886c-2d247a538bed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:38:13.094427 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:13.094404 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/101eb7b8-0e9e-4798-886c-2d247a538bed-kube-api-access-nrkxd" (OuterVolumeSpecName: "kube-api-access-nrkxd") pod "101eb7b8-0e9e-4798-886c-2d247a538bed" (UID: "101eb7b8-0e9e-4798-886c-2d247a538bed"). InnerVolumeSpecName "kube-api-access-nrkxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:38:13.094495 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:13.094431 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/101eb7b8-0e9e-4798-886c-2d247a538bed-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "101eb7b8-0e9e-4798-886c-2d247a538bed" (UID: "101eb7b8-0e9e-4798-886c-2d247a538bed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:38:13.193717 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:13.193686 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrkxd\" (UniqueName: \"kubernetes.io/projected/101eb7b8-0e9e-4798-886c-2d247a538bed-kube-api-access-nrkxd\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:38:13.193717 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:13.193710 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/101eb7b8-0e9e-4798-886c-2d247a538bed-maas-api-tls\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:38:13.193717 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:13.193720 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/101eb7b8-0e9e-4798-886c-2d247a538bed-openshift-service-ca\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:38:13.594302 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:13.594217 2558 generic.go:356] "Generic (PLEG): container finished" podID="101eb7b8-0e9e-4798-886c-2d247a538bed" containerID="43d776b272fd1968f2b310cb326633d2e234c1e93df4ca5e2598820294dcfdb8" exitCode=0 Jul 08 19:38:13.594302 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:13.594267 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5f8b33-754f4c77db-pw97v" Jul 08 19:38:13.594483 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:13.594306 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5f8b33-754f4c77db-pw97v" event={"ID":"101eb7b8-0e9e-4798-886c-2d247a538bed","Type":"ContainerDied","Data":"43d776b272fd1968f2b310cb326633d2e234c1e93df4ca5e2598820294dcfdb8"} Jul 08 19:38:13.594483 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:13.594381 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5f8b33-754f4c77db-pw97v" event={"ID":"101eb7b8-0e9e-4798-886c-2d247a538bed","Type":"ContainerDied","Data":"4329fe7c1cc32c3f8fcc524cb176d9c3730c9a4b708aa10840d0109556590458"} Jul 08 19:38:13.594483 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:13.594404 2558 scope.go:117] "RemoveContainer" containerID="43d776b272fd1968f2b310cb326633d2e234c1e93df4ca5e2598820294dcfdb8" Jul 08 19:38:13.603408 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:13.603390 2558 scope.go:117] "RemoveContainer" containerID="43d776b272fd1968f2b310cb326633d2e234c1e93df4ca5e2598820294dcfdb8" Jul 08 19:38:13.603725 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:38:13.603705 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43d776b272fd1968f2b310cb326633d2e234c1e93df4ca5e2598820294dcfdb8\": container with ID starting with 43d776b272fd1968f2b310cb326633d2e234c1e93df4ca5e2598820294dcfdb8 not found: ID does not exist" containerID="43d776b272fd1968f2b310cb326633d2e234c1e93df4ca5e2598820294dcfdb8" Jul 08 19:38:13.603800 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:13.603733 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43d776b272fd1968f2b310cb326633d2e234c1e93df4ca5e2598820294dcfdb8"} err="failed to get container status \"43d776b272fd1968f2b310cb326633d2e234c1e93df4ca5e2598820294dcfdb8\": rpc error: code = NotFound desc = could not find container \"43d776b272fd1968f2b310cb326633d2e234c1e93df4ca5e2598820294dcfdb8\": container with ID starting with 43d776b272fd1968f2b310cb326633d2e234c1e93df4ca5e2598820294dcfdb8 not found: ID does not exist" Jul 08 19:38:13.612084 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:13.612061 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5f8b33-754f4c77db-pw97v"] Jul 08 19:38:13.616741 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:13.616716 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5f8b33-754f4c77db-pw97v"] Jul 08 19:38:15.282989 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:15.282957 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="101eb7b8-0e9e-4798-886c-2d247a538bed" path="/var/lib/kubelet/pods/101eb7b8-0e9e-4798-886c-2d247a538bed/volumes" Jul 08 19:38:18.330976 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:18.330946 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:18.393374 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:18.393346 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:18.648599 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:18.648539 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:19.054392 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:19.054311 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:19.469288 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:19.469259 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:19.573441 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:19.573406 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:19.673508 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:19.673466 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:19.919431 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:19.919387 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-39e11a-676f6b9598-vbgb2"] Jul 08 19:38:19.919780 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:19.919749 2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e11a-676f6b9598-vbgb2" podUID="3ccd7e0d-b383-4203-9643-f01894ba1810" containerName="maas-api" containerID="cri-o://076cb19e7281fad2065ebe7cff2641a960622aaa53fc09d59114b7dbf142389c" gracePeriod=30 Jul 08 19:38:19.957870 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:19.957847 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:20.177721 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:20.177658 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e11a-676f6b9598-vbgb2" Jul 08 19:38:20.258273 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:20.258246 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ccd7e0d-b383-4203-9643-f01894ba1810-openshift-service-ca\") pod \"3ccd7e0d-b383-4203-9643-f01894ba1810\" (UID: \"3ccd7e0d-b383-4203-9643-f01894ba1810\") " Jul 08 19:38:20.258273 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:20.258275 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ccd7e0d-b383-4203-9643-f01894ba1810-maas-api-tls\") pod \"3ccd7e0d-b383-4203-9643-f01894ba1810\" (UID: \"3ccd7e0d-b383-4203-9643-f01894ba1810\") " Jul 08 19:38:20.258450 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:20.258296 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zm8db\" (UniqueName: \"kubernetes.io/projected/3ccd7e0d-b383-4203-9643-f01894ba1810-kube-api-access-zm8db\") pod \"3ccd7e0d-b383-4203-9643-f01894ba1810\" (UID: \"3ccd7e0d-b383-4203-9643-f01894ba1810\") " Jul 08 19:38:20.258629 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:20.258604 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ccd7e0d-b383-4203-9643-f01894ba1810-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3ccd7e0d-b383-4203-9643-f01894ba1810" (UID: "3ccd7e0d-b383-4203-9643-f01894ba1810"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:38:20.260346 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:20.260315 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ccd7e0d-b383-4203-9643-f01894ba1810-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3ccd7e0d-b383-4203-9643-f01894ba1810" (UID: "3ccd7e0d-b383-4203-9643-f01894ba1810"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:38:20.260476 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:20.260358 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ccd7e0d-b383-4203-9643-f01894ba1810-kube-api-access-zm8db" (OuterVolumeSpecName: "kube-api-access-zm8db") pod "3ccd7e0d-b383-4203-9643-f01894ba1810" (UID: "3ccd7e0d-b383-4203-9643-f01894ba1810"). InnerVolumeSpecName "kube-api-access-zm8db". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:38:20.359297 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:20.359247 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ccd7e0d-b383-4203-9643-f01894ba1810-openshift-service-ca\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:38:20.359297 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:20.359292 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ccd7e0d-b383-4203-9643-f01894ba1810-maas-api-tls\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:38:20.359297 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:20.359305 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm8db\" (UniqueName: \"kubernetes.io/projected/3ccd7e0d-b383-4203-9643-f01894ba1810-kube-api-access-zm8db\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:38:20.639388 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:20.639339 2558 generic.go:356] "Generic (PLEG): container finished" podID="3ccd7e0d-b383-4203-9643-f01894ba1810" containerID="076cb19e7281fad2065ebe7cff2641a960622aaa53fc09d59114b7dbf142389c" exitCode=0 Jul 08 19:38:20.639388 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:20.639377 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e11a-676f6b9598-vbgb2" event={"ID":"3ccd7e0d-b383-4203-9643-f01894ba1810","Type":"ContainerDied","Data":"076cb19e7281fad2065ebe7cff2641a960622aaa53fc09d59114b7dbf142389c"} Jul 08 19:38:20.639962 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:20.639438 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e11a-676f6b9598-vbgb2" event={"ID":"3ccd7e0d-b383-4203-9643-f01894ba1810","Type":"ContainerDied","Data":"4efaeca2876fd61696d82cbd83f01f5c4e3768a037da1743363182b64a7e7146"} Jul 08 19:38:20.639962 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:20.639469 2558 scope.go:117] "RemoveContainer" containerID="076cb19e7281fad2065ebe7cff2641a960622aaa53fc09d59114b7dbf142389c" Jul 08 19:38:20.639962 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:20.639500 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-39e11a-676f6b9598-vbgb2" Jul 08 19:38:20.649673 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:20.649652 2558 scope.go:117] "RemoveContainer" containerID="076cb19e7281fad2065ebe7cff2641a960622aaa53fc09d59114b7dbf142389c" Jul 08 19:38:20.649938 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:38:20.649919 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"076cb19e7281fad2065ebe7cff2641a960622aaa53fc09d59114b7dbf142389c\": container with ID starting with 076cb19e7281fad2065ebe7cff2641a960622aaa53fc09d59114b7dbf142389c not found: ID does not exist" containerID="076cb19e7281fad2065ebe7cff2641a960622aaa53fc09d59114b7dbf142389c" Jul 08 19:38:20.650013 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:20.649946 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"076cb19e7281fad2065ebe7cff2641a960622aaa53fc09d59114b7dbf142389c"} err="failed to get container status \"076cb19e7281fad2065ebe7cff2641a960622aaa53fc09d59114b7dbf142389c\": rpc error: code = NotFound desc = could not find container \"076cb19e7281fad2065ebe7cff2641a960622aaa53fc09d59114b7dbf142389c\": container with ID starting with 076cb19e7281fad2065ebe7cff2641a960622aaa53fc09d59114b7dbf142389c not found: ID does not exist" Jul 08 19:38:20.663650 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:20.663625 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-39e11a-676f6b9598-vbgb2"] Jul 08 19:38:20.665714 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:20.665691 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-39e11a-676f6b9598-vbgb2"] Jul 08 19:38:21.281681 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:21.281649 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ccd7e0d-b383-4203-9643-f01894ba1810" path="/var/lib/kubelet/pods/3ccd7e0d-b383-4203-9643-f01894ba1810/volumes" Jul 08 19:38:25.583121 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:25.583080 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:25.683015 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:25.682976 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:25.876001 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:25.875969 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:26.443354 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:26.443320 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:26.769350 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:26.769251 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:27.105229 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:27.105139 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:29.457345 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:29.457309 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:29.558447 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:29.558414 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:29.559104 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:29.559081 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d86987-6755859574-z98zr"] Jul 08 19:38:29.559421 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:29.559410 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="101eb7b8-0e9e-4798-886c-2d247a538bed" containerName="maas-api" Jul 08 19:38:29.559459 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:29.559422 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="101eb7b8-0e9e-4798-886c-2d247a538bed" containerName="maas-api" Jul 08 19:38:29.559459 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:29.559436 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ccd7e0d-b383-4203-9643-f01894ba1810" containerName="maas-api" Jul 08 19:38:29.559459 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:29.559441 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ccd7e0d-b383-4203-9643-f01894ba1810" containerName="maas-api" Jul 08 19:38:29.559604 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:29.559508 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="101eb7b8-0e9e-4798-886c-2d247a538bed" containerName="maas-api" Jul 08 19:38:29.559604 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:29.559535 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ccd7e0d-b383-4203-9643-f01894ba1810" containerName="maas-api" Jul 08 19:38:29.563930 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:29.563913 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d86987-6755859574-z98zr" Jul 08 19:38:29.567038 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:29.567017 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-d86987-serving-cert\"" Jul 08 19:38:29.577687 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:29.577657 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d86987-6755859574-z98zr"] Jul 08 19:38:29.624197 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:29.624163 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:29.630546 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:29.630524 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02739959-e191-436d-9c31-8388093e9501-maas-api-tls\") pod \"maas-api-e2e-shared-a-d86987-6755859574-z98zr\" (UID: \"02739959-e191-436d-9c31-8388093e9501\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d86987-6755859574-z98zr" Jul 08 19:38:29.630642 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:29.630558 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hfhh\" (UniqueName: \"kubernetes.io/projected/02739959-e191-436d-9c31-8388093e9501-kube-api-access-8hfhh\") pod \"maas-api-e2e-shared-a-d86987-6755859574-z98zr\" (UID: \"02739959-e191-436d-9c31-8388093e9501\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d86987-6755859574-z98zr" Jul 08 19:38:29.630642 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:29.630604 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02739959-e191-436d-9c31-8388093e9501-openshift-service-ca\") pod \"maas-api-e2e-shared-a-d86987-6755859574-z98zr\" (UID: \"02739959-e191-436d-9c31-8388093e9501\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d86987-6755859574-z98zr" Jul 08 19:38:29.731342 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:29.731256 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02739959-e191-436d-9c31-8388093e9501-maas-api-tls\") pod \"maas-api-e2e-shared-a-d86987-6755859574-z98zr\" (UID: \"02739959-e191-436d-9c31-8388093e9501\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d86987-6755859574-z98zr" Jul 08 19:38:29.731342 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:29.731293 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hfhh\" (UniqueName: \"kubernetes.io/projected/02739959-e191-436d-9c31-8388093e9501-kube-api-access-8hfhh\") pod \"maas-api-e2e-shared-a-d86987-6755859574-z98zr\" (UID: \"02739959-e191-436d-9c31-8388093e9501\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d86987-6755859574-z98zr" Jul 08 19:38:29.731342 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:29.731319 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02739959-e191-436d-9c31-8388093e9501-openshift-service-ca\") pod \"maas-api-e2e-shared-a-d86987-6755859574-z98zr\" (UID: \"02739959-e191-436d-9c31-8388093e9501\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d86987-6755859574-z98zr" Jul 08 19:38:29.731613 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:38:29.731468 2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-d86987-serving-cert: secret "maas-api-e2e-shared-a-d86987-serving-cert" not found Jul 08 19:38:29.731613 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:38:29.731569 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02739959-e191-436d-9c31-8388093e9501-maas-api-tls podName:02739959-e191-436d-9c31-8388093e9501 nodeName:}" failed. No retries permitted until 2026-07-08 19:38:30.231548641 +0000 UTC m=+2093.490501553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/02739959-e191-436d-9c31-8388093e9501-maas-api-tls") pod "maas-api-e2e-shared-a-d86987-6755859574-z98zr" (UID: "02739959-e191-436d-9c31-8388093e9501") : secret "maas-api-e2e-shared-a-d86987-serving-cert" not found Jul 08 19:38:29.732071 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:29.732049 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02739959-e191-436d-9c31-8388093e9501-openshift-service-ca\") pod \"maas-api-e2e-shared-a-d86987-6755859574-z98zr\" (UID: \"02739959-e191-436d-9c31-8388093e9501\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d86987-6755859574-z98zr" Jul 08 19:38:29.740100 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:29.740073 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hfhh\" (UniqueName: \"kubernetes.io/projected/02739959-e191-436d-9c31-8388093e9501-kube-api-access-8hfhh\") pod \"maas-api-e2e-shared-a-d86987-6755859574-z98zr\" (UID: \"02739959-e191-436d-9c31-8388093e9501\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d86987-6755859574-z98zr" Jul 08 19:38:29.851072 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:29.851038 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:30.136561 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:30.136524 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:30.232230 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:30.232201 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:30.235206 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:30.235184 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02739959-e191-436d-9c31-8388093e9501-maas-api-tls\") pod \"maas-api-e2e-shared-a-d86987-6755859574-z98zr\" (UID: \"02739959-e191-436d-9c31-8388093e9501\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d86987-6755859574-z98zr" Jul 08 19:38:30.237572 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:30.237545 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02739959-e191-436d-9c31-8388093e9501-maas-api-tls\") pod \"maas-api-e2e-shared-a-d86987-6755859574-z98zr\" (UID: \"02739959-e191-436d-9c31-8388093e9501\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d86987-6755859574-z98zr" Jul 08 19:38:30.474548 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:30.474466 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d86987-6755859574-z98zr" Jul 08 19:38:30.614382 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:30.614349 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d86987-6755859574-z98zr"] Jul 08 19:38:30.615538 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:38:30.615512 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02739959_e191_436d_9c31_8388093e9501.slice/crio-6dd0326ea128bf83440bdcc20610a10ed2afd051f7e8ea70a5f6a4f6982047b3 WatchSource:0}: Error finding container 6dd0326ea128bf83440bdcc20610a10ed2afd051f7e8ea70a5f6a4f6982047b3: Status 404 returned error can't find the container with id 6dd0326ea128bf83440bdcc20610a10ed2afd051f7e8ea70a5f6a4f6982047b3 Jul 08 19:38:30.694470 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:30.694437 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d86987-6755859574-z98zr" event={"ID":"02739959-e191-436d-9c31-8388093e9501","Type":"ContainerStarted","Data":"6dd0326ea128bf83440bdcc20610a10ed2afd051f7e8ea70a5f6a4f6982047b3"} Jul 08 19:38:31.701017 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:31.700970 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d86987-6755859574-z98zr" event={"ID":"02739959-e191-436d-9c31-8388093e9501","Type":"ContainerStarted","Data":"b40fc666b7bcb1342d321107a011d1656284b68e86ef11ed8658866ac818a314"} Jul 08 19:38:31.701402 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:31.701105 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d86987-6755859574-z98zr" Jul 08 19:38:31.719397 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:31.719347 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d86987-6755859574-z98zr" podStartSLOduration=2.401577318 podStartE2EDuration="2.719333826s" totalImagesPullingTime="317.756508ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:38:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:38:31.718379266 +0000 UTC m=+2094.977332190" watchObservedRunningTime="2026-07-08 19:38:31.719333826 +0000 UTC m=+2094.978286748" Jul 08 19:38:33.944789 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:33.944741 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:34.285599 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:34.285497 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:34.593980 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:34.593900 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:34.871511 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:34.871471 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:37.361146 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:37.361114 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-54hnr_1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a/console-operator/2.log" Jul 08 19:38:37.363183 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:37.363160 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-54hnr_1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a/console-operator/2.log" Jul 08 19:38:37.675504 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:37.675456 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:37.710987 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:37.710955 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d86987-6755859574-z98zr" Jul 08 19:38:37.771075 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:37.771031 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:37.925112 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:37.925081 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4a9cb4-7bb87f5448-rngth"] Jul 08 19:38:37.929997 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:37.929938 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4a9cb4-7bb87f5448-rngth" Jul 08 19:38:37.933778 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:37.933756 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-4a9cb4-serving-cert\"" Jul 08 19:38:37.953082 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:37.953054 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4a9cb4-7bb87f5448-rngth"] Jul 08 19:38:37.982563 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:37.982528 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:38.000785 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:38.000758 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/433dc1c4-8c47-4f00-bc80-518fff9ca429-openshift-service-ca\") pod \"maas-api-e2e-shared-b-4a9cb4-7bb87f5448-rngth\" (UID: \"433dc1c4-8c47-4f00-bc80-518fff9ca429\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4a9cb4-7bb87f5448-rngth" Jul 08 19:38:38.000928 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:38.000793 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prz54\" (UniqueName: \"kubernetes.io/projected/433dc1c4-8c47-4f00-bc80-518fff9ca429-kube-api-access-prz54\") pod \"maas-api-e2e-shared-b-4a9cb4-7bb87f5448-rngth\" (UID: \"433dc1c4-8c47-4f00-bc80-518fff9ca429\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4a9cb4-7bb87f5448-rngth" Jul 08 19:38:38.000928 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:38.000831 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/433dc1c4-8c47-4f00-bc80-518fff9ca429-maas-api-tls\") pod \"maas-api-e2e-shared-b-4a9cb4-7bb87f5448-rngth\" (UID: \"433dc1c4-8c47-4f00-bc80-518fff9ca429\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4a9cb4-7bb87f5448-rngth" Jul 08 19:38:38.102267 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:38.102228 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/433dc1c4-8c47-4f00-bc80-518fff9ca429-openshift-service-ca\") pod \"maas-api-e2e-shared-b-4a9cb4-7bb87f5448-rngth\" (UID: \"433dc1c4-8c47-4f00-bc80-518fff9ca429\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4a9cb4-7bb87f5448-rngth" Jul 08 19:38:38.102267 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:38.102264 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prz54\" (UniqueName: \"kubernetes.io/projected/433dc1c4-8c47-4f00-bc80-518fff9ca429-kube-api-access-prz54\") pod \"maas-api-e2e-shared-b-4a9cb4-7bb87f5448-rngth\" (UID: \"433dc1c4-8c47-4f00-bc80-518fff9ca429\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4a9cb4-7bb87f5448-rngth" Jul 08 19:38:38.102520 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:38.102287 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/433dc1c4-8c47-4f00-bc80-518fff9ca429-maas-api-tls\") pod \"maas-api-e2e-shared-b-4a9cb4-7bb87f5448-rngth\" (UID: \"433dc1c4-8c47-4f00-bc80-518fff9ca429\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4a9cb4-7bb87f5448-rngth" Jul 08 19:38:38.102520 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:38:38.102421 2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-4a9cb4-serving-cert: secret "maas-api-e2e-shared-b-4a9cb4-serving-cert" not found Jul 08 19:38:38.102520 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:38:38.102489 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/433dc1c4-8c47-4f00-bc80-518fff9ca429-maas-api-tls podName:433dc1c4-8c47-4f00-bc80-518fff9ca429 nodeName:}" failed. No retries permitted until 2026-07-08 19:38:38.602469546 +0000 UTC m=+2101.861422466 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/433dc1c4-8c47-4f00-bc80-518fff9ca429-maas-api-tls") pod "maas-api-e2e-shared-b-4a9cb4-7bb87f5448-rngth" (UID: "433dc1c4-8c47-4f00-bc80-518fff9ca429") : secret "maas-api-e2e-shared-b-4a9cb4-serving-cert" not found Jul 08 19:38:38.102996 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:38.102972 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/433dc1c4-8c47-4f00-bc80-518fff9ca429-openshift-service-ca\") pod \"maas-api-e2e-shared-b-4a9cb4-7bb87f5448-rngth\" (UID: \"433dc1c4-8c47-4f00-bc80-518fff9ca429\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4a9cb4-7bb87f5448-rngth" Jul 08 19:38:38.114946 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:38.114923 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prz54\" (UniqueName: \"kubernetes.io/projected/433dc1c4-8c47-4f00-bc80-518fff9ca429-kube-api-access-prz54\") pod \"maas-api-e2e-shared-b-4a9cb4-7bb87f5448-rngth\" (UID: \"433dc1c4-8c47-4f00-bc80-518fff9ca429\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4a9cb4-7bb87f5448-rngth" Jul 08 19:38:38.589119 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:38.589081 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:38.607556 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:38.607529 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/433dc1c4-8c47-4f00-bc80-518fff9ca429-maas-api-tls\") pod \"maas-api-e2e-shared-b-4a9cb4-7bb87f5448-rngth\" (UID: \"433dc1c4-8c47-4f00-bc80-518fff9ca429\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4a9cb4-7bb87f5448-rngth" Jul 08 19:38:38.609995 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:38.609969 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/433dc1c4-8c47-4f00-bc80-518fff9ca429-maas-api-tls\") pod \"maas-api-e2e-shared-b-4a9cb4-7bb87f5448-rngth\" (UID: \"433dc1c4-8c47-4f00-bc80-518fff9ca429\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4a9cb4-7bb87f5448-rngth" Jul 08 19:38:38.688192 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:38.688158 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:38.800619 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:38.800565 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:38.839780 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:38.839688 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4a9cb4-7bb87f5448-rngth" Jul 08 19:38:38.986207 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:38.986172 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4a9cb4-7bb87f5448-rngth"] Jul 08 19:38:38.987257 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:38:38.987232 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod433dc1c4_8c47_4f00_bc80_518fff9ca429.slice/crio-34efc2d058ad50e6b66e24d5bfbece5c8ed195d46e322f18acc637855ae39233 WatchSource:0}: Error finding container 34efc2d058ad50e6b66e24d5bfbece5c8ed195d46e322f18acc637855ae39233: Status 404 returned error can't find the container with id 34efc2d058ad50e6b66e24d5bfbece5c8ed195d46e322f18acc637855ae39233 Jul 08 19:38:39.743143 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:39.743048 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4a9cb4-7bb87f5448-rngth" event={"ID":"433dc1c4-8c47-4f00-bc80-518fff9ca429","Type":"ContainerStarted","Data":"b0f38d5c94eaa2617b10d608b535e4be4926f2859c46329508acb9434da2a2d4"} Jul 08 19:38:39.743143 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:39.743105 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4a9cb4-7bb87f5448-rngth" Jul 08 19:38:39.743143 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:39.743122 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4a9cb4-7bb87f5448-rngth" event={"ID":"433dc1c4-8c47-4f00-bc80-518fff9ca429","Type":"ContainerStarted","Data":"34efc2d058ad50e6b66e24d5bfbece5c8ed195d46e322f18acc637855ae39233"} Jul 08 19:38:39.765282 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:39.765231 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4a9cb4-7bb87f5448-rngth" podStartSLOduration=2.310316165 podStartE2EDuration="2.76521459s" totalImagesPullingTime="454.898425ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:38:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:38:39.763121973 +0000 UTC m=+2103.022074898" watchObservedRunningTime="2026-07-08 19:38:39.76521459 +0000 UTC m=+2103.024167513" Jul 08 19:38:42.076569 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:42.076530 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:45.754495 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:45.754462 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4a9cb4-7bb87f5448-rngth" Jul 08 19:38:50.993019 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:50.992980 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:51.093359 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:51.093318 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:59.459284 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:59.459248 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:38:59.616148 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:38:59.616099 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:39:04.837964 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:39:04.837927 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:39:22.131325 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:39:22.131283 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:40:36.425161 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:40:36.425124 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:41:25.045235 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:41:25.045184 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:41:25.935798 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:41:25.935757 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:41:33.640347 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:41:33.640304 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:42:08.733951 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:42:08.733912 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:42:29.070809 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:42:29.070770 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:42:35.017016 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:42:35.016972 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:42:48.621172 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:42:48.621132 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:42:48.723470 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:42:48.723432 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:42:49.377297 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:42:49.377249 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:42:50.424466 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:42:50.424423 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:42:59.505865 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:42:59.505811 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:43:18.261072 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:43:18.261033 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:43:22.409199 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:43:22.409161 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:43:37.394149 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:43:37.394119 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-54hnr_1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a/console-operator/2.log" Jul 08 19:43:37.398991 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:43:37.398969 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-54hnr_1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a/console-operator/2.log" Jul 08 19:43:47.239833 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:43:47.239794 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:43:51.704451 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:43:51.704410 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:43:51.810431 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:43:51.810395 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:44:00.176539 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:44:00.176423 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:44:00.276917 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:44:00.276861 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:44:31.055627 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:44:31.055573 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:44:32.128952 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:44:32.128908 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:45:00.137951 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:00.137911 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4a9cb4-29725665-mpw8m"] Jul 08 19:45:00.141569 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:00.141548 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d86987-29725665-2srfn"] Jul 08 19:45:00.141720 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:00.141697 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4a9cb4-29725665-mpw8m" Jul 08 19:45:00.144733 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:00.144716 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725665-z6bgf"] Jul 08 19:45:00.144855 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:00.144834 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d86987-29725665-2srfn" Jul 08 19:45:00.148302 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:00.148274 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725665-z6bgf" Jul 08 19:45:00.149205 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:00.149146 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4a9cb4-29725665-mpw8m"] Jul 08 19:45:00.157968 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:00.156681 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d86987-29725665-2srfn"] Jul 08 19:45:00.161480 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:00.161457 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725665-z6bgf"] Jul 08 19:45:00.258358 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:00.258321 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrdkb\" (UniqueName: \"kubernetes.io/projected/7840aff8-3d34-4255-978f-4d9d299be7e8-kube-api-access-xrdkb\") pod \"maas-api-key-cleanup-e2e-shared-b-4a9cb4-29725665-mpw8m\" (UID: \"7840aff8-3d34-4255-978f-4d9d299be7e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4a9cb4-29725665-mpw8m" Jul 08 19:45:00.258509 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:00.258365 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzkvx\" (UniqueName: \"kubernetes.io/projected/b2bbf45f-e9d9-4548-83b4-07280d39dc05-kube-api-access-jzkvx\") pod \"maas-api-key-cleanup-29725665-z6bgf\" (UID: \"b2bbf45f-e9d9-4548-83b4-07280d39dc05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725665-z6bgf" Jul 08 19:45:00.258509 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:00.258433 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt5hd\" (UniqueName: \"kubernetes.io/projected/891db0cb-c6a4-4cac-a0c7-6bca3a4711d6-kube-api-access-nt5hd\") pod \"maas-api-key-cleanup-e2e-shared-a-d86987-29725665-2srfn\" (UID: \"891db0cb-c6a4-4cac-a0c7-6bca3a4711d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d86987-29725665-2srfn" Jul 08 19:45:00.358897 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:00.358862 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt5hd\" (UniqueName: \"kubernetes.io/projected/891db0cb-c6a4-4cac-a0c7-6bca3a4711d6-kube-api-access-nt5hd\") pod \"maas-api-key-cleanup-e2e-shared-a-d86987-29725665-2srfn\" (UID: \"891db0cb-c6a4-4cac-a0c7-6bca3a4711d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d86987-29725665-2srfn" Jul 08 19:45:00.359088 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:00.359062 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrdkb\" (UniqueName: \"kubernetes.io/projected/7840aff8-3d34-4255-978f-4d9d299be7e8-kube-api-access-xrdkb\") pod \"maas-api-key-cleanup-e2e-shared-b-4a9cb4-29725665-mpw8m\" (UID: \"7840aff8-3d34-4255-978f-4d9d299be7e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4a9cb4-29725665-mpw8m" Jul 08 19:45:00.359234 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:00.359211 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzkvx\" (UniqueName: \"kubernetes.io/projected/b2bbf45f-e9d9-4548-83b4-07280d39dc05-kube-api-access-jzkvx\") pod \"maas-api-key-cleanup-29725665-z6bgf\" (UID: \"b2bbf45f-e9d9-4548-83b4-07280d39dc05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725665-z6bgf" Jul 08 19:45:00.366901 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:00.366880 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt5hd\" (UniqueName: \"kubernetes.io/projected/891db0cb-c6a4-4cac-a0c7-6bca3a4711d6-kube-api-access-nt5hd\") pod \"maas-api-key-cleanup-e2e-shared-a-d86987-29725665-2srfn\" (UID: \"891db0cb-c6a4-4cac-a0c7-6bca3a4711d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d86987-29725665-2srfn" Jul 08 19:45:00.367012 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:00.366915 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrdkb\" (UniqueName: \"kubernetes.io/projected/7840aff8-3d34-4255-978f-4d9d299be7e8-kube-api-access-xrdkb\") pod \"maas-api-key-cleanup-e2e-shared-b-4a9cb4-29725665-mpw8m\" (UID: \"7840aff8-3d34-4255-978f-4d9d299be7e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4a9cb4-29725665-mpw8m" Jul 08 19:45:00.367012 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:00.367002 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzkvx\" (UniqueName: \"kubernetes.io/projected/b2bbf45f-e9d9-4548-83b4-07280d39dc05-kube-api-access-jzkvx\") pod \"maas-api-key-cleanup-29725665-z6bgf\" (UID: \"b2bbf45f-e9d9-4548-83b4-07280d39dc05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725665-z6bgf" Jul 08 19:45:00.461418 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:00.461338 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4a9cb4-29725665-mpw8m" Jul 08 19:45:00.469445 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:00.469424 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d86987-29725665-2srfn" Jul 08 19:45:00.476901 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:00.476718 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725665-z6bgf" Jul 08 19:45:00.618719 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:00.618686 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4a9cb4-29725665-mpw8m"] Jul 08 19:45:00.619307 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:45:00.619278 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7840aff8_3d34_4255_978f_4d9d299be7e8.slice/crio-3168f0d9e29efeacf0afea2f48d58abc29affdb6ff29504fa49b686c8e42da43 WatchSource:0}: Error finding container 3168f0d9e29efeacf0afea2f48d58abc29affdb6ff29504fa49b686c8e42da43: Status 404 returned error can't find the container with id 3168f0d9e29efeacf0afea2f48d58abc29affdb6ff29504fa49b686c8e42da43 Jul 08 19:45:00.699653 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:00.699558 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4a9cb4-29725665-mpw8m" event={"ID":"7840aff8-3d34-4255-978f-4d9d299be7e8","Type":"ContainerStarted","Data":"3168f0d9e29efeacf0afea2f48d58abc29affdb6ff29504fa49b686c8e42da43"} Jul 08 19:45:00.850780 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:00.850752 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d86987-29725665-2srfn"] Jul 08 19:45:00.851784 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:45:00.851757 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod891db0cb_c6a4_4cac_a0c7_6bca3a4711d6.slice/crio-0f935a82acdfd00bc66da25965a471a381ed097deb7569a8cc7dd37c82331976 WatchSource:0}: Error finding container 0f935a82acdfd00bc66da25965a471a381ed097deb7569a8cc7dd37c82331976: Status 404 returned error can't find the container with id 0f935a82acdfd00bc66da25965a471a381ed097deb7569a8cc7dd37c82331976 Jul 08 19:45:00.853494 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:00.853469 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725665-z6bgf"] Jul 08 19:45:00.854791 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:45:00.854767 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2bbf45f_e9d9_4548_83b4_07280d39dc05.slice/crio-874975fc0f4e3e2ee6696e44329c3379a0cb62938f40878bcc5732d462457d35 WatchSource:0}: Error finding container 874975fc0f4e3e2ee6696e44329c3379a0cb62938f40878bcc5732d462457d35: Status 404 returned error can't find the container with id 874975fc0f4e3e2ee6696e44329c3379a0cb62938f40878bcc5732d462457d35 Jul 08 19:45:01.705860 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:01.705824 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4a9cb4-29725665-mpw8m" event={"ID":"7840aff8-3d34-4255-978f-4d9d299be7e8","Type":"ContainerStarted","Data":"459ff971a025e3149e99964a23f44d6e17b0e98924bee977543b124b159e7758"} Jul 08 19:45:01.707303 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:01.707270 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d86987-29725665-2srfn" event={"ID":"891db0cb-c6a4-4cac-a0c7-6bca3a4711d6","Type":"ContainerStarted","Data":"cf366d5d8237890f612bf50369d190ddc67311186c8aa36e6a6d2fb24f4b8301"} Jul 08 19:45:01.707436 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:01.707317 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d86987-29725665-2srfn" event={"ID":"891db0cb-c6a4-4cac-a0c7-6bca3a4711d6","Type":"ContainerStarted","Data":"0f935a82acdfd00bc66da25965a471a381ed097deb7569a8cc7dd37c82331976"} Jul 08 19:45:01.708673 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:01.708653 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725665-z6bgf" event={"ID":"b2bbf45f-e9d9-4548-83b4-07280d39dc05","Type":"ContainerStarted","Data":"05373a9607672edc87cdd6fcf169da1b0dd535a5453acdb867e42c6ef4515832"} Jul 08 19:45:01.708785 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:01.708686 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725665-z6bgf" event={"ID":"b2bbf45f-e9d9-4548-83b4-07280d39dc05","Type":"ContainerStarted","Data":"874975fc0f4e3e2ee6696e44329c3379a0cb62938f40878bcc5732d462457d35"} Jul 08 19:45:01.723424 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:01.723371 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4a9cb4-29725665-mpw8m" podStartSLOduration=1.723356811 podStartE2EDuration="1.723356811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:45:01.720882085 +0000 UTC m=+2484.979835009" watchObservedRunningTime="2026-07-08 19:45:01.723356811 +0000 UTC m=+2484.982309737" Jul 08 19:45:01.736527 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:01.736483 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d86987-29725665-2srfn" podStartSLOduration=1.736470661 podStartE2EDuration="1.736470661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:45:01.735604342 +0000 UTC m=+2484.994557269" watchObservedRunningTime="2026-07-08 19:45:01.736470661 +0000 UTC m=+2484.995423584" Jul 08 19:45:01.751795 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:01.751749 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725665-z6bgf" podStartSLOduration=1.751736888 podStartE2EDuration="1.751736888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:45:01.750695077 +0000 UTC m=+2485.009648001" watchObservedRunningTime="2026-07-08 19:45:01.751736888 +0000 UTC m=+2485.010689825" Jul 08 19:45:12.468351 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:12.468305 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:45:12.968947 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:12.968909 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:45:13.066819 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:13.066783 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:45:13.190305 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:13.190268 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d86987-6755859574-z98zr"] Jul 08 19:45:13.190512 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:13.190485 2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d86987-6755859574-z98zr" podUID="02739959-e191-436d-9c31-8388093e9501" containerName="maas-api" containerID="cri-o://b40fc666b7bcb1342d321107a011d1656284b68e86ef11ed8658866ac818a314" gracePeriod=30 Jul 08 19:45:13.218555 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:13.218524 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d86987-29725665-2srfn"] Jul 08 19:45:13.218809 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:13.218769 2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d86987-29725665-2srfn" podUID="891db0cb-c6a4-4cac-a0c7-6bca3a4711d6" containerName="cleanup" containerID="cri-o://cf366d5d8237890f612bf50369d190ddc67311186c8aa36e6a6d2fb24f4b8301" gracePeriod=30 Jul 08 19:45:13.220869 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:13.220841 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:45:13.644077 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:13.644051 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d86987-6755859574-z98zr" Jul 08 19:45:13.773828 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:13.773738 2558 generic.go:356] "Generic (PLEG): container finished" podID="02739959-e191-436d-9c31-8388093e9501" containerID="b40fc666b7bcb1342d321107a011d1656284b68e86ef11ed8658866ac818a314" exitCode=0 Jul 08 19:45:13.773828 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:13.773785 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d86987-6755859574-z98zr" event={"ID":"02739959-e191-436d-9c31-8388093e9501","Type":"ContainerDied","Data":"b40fc666b7bcb1342d321107a011d1656284b68e86ef11ed8658866ac818a314"} Jul 08 19:45:13.774008 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:13.773848 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d86987-6755859574-z98zr" event={"ID":"02739959-e191-436d-9c31-8388093e9501","Type":"ContainerDied","Data":"6dd0326ea128bf83440bdcc20610a10ed2afd051f7e8ea70a5f6a4f6982047b3"} Jul 08 19:45:13.774008 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:13.773874 2558 scope.go:117] "RemoveContainer" containerID="b40fc666b7bcb1342d321107a011d1656284b68e86ef11ed8658866ac818a314" Jul 08 19:45:13.774008 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:13.773796 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d86987-6755859574-z98zr" Jul 08 19:45:13.776939 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:13.776918 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02739959-e191-436d-9c31-8388093e9501-openshift-service-ca\") pod \"02739959-e191-436d-9c31-8388093e9501\" (UID: \"02739959-e191-436d-9c31-8388093e9501\") " Jul 08 19:45:13.777058 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:13.777009 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hfhh\" (UniqueName: \"kubernetes.io/projected/02739959-e191-436d-9c31-8388093e9501-kube-api-access-8hfhh\") pod \"02739959-e191-436d-9c31-8388093e9501\" (UID: \"02739959-e191-436d-9c31-8388093e9501\") " Jul 08 19:45:13.777121 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:13.777057 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02739959-e191-436d-9c31-8388093e9501-maas-api-tls\") pod \"02739959-e191-436d-9c31-8388093e9501\" (UID: \"02739959-e191-436d-9c31-8388093e9501\") " Jul 08 19:45:13.777364 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:13.777337 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02739959-e191-436d-9c31-8388093e9501-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "02739959-e191-436d-9c31-8388093e9501" (UID: "02739959-e191-436d-9c31-8388093e9501"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:45:13.779108 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:13.779087 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02739959-e191-436d-9c31-8388093e9501-kube-api-access-8hfhh" (OuterVolumeSpecName: "kube-api-access-8hfhh") pod "02739959-e191-436d-9c31-8388093e9501" (UID: "02739959-e191-436d-9c31-8388093e9501"). InnerVolumeSpecName "kube-api-access-8hfhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:45:13.779370 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:13.779349 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02739959-e191-436d-9c31-8388093e9501-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "02739959-e191-436d-9c31-8388093e9501" (UID: "02739959-e191-436d-9c31-8388093e9501"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:45:13.783252 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:13.783236 2558 scope.go:117] "RemoveContainer" containerID="b40fc666b7bcb1342d321107a011d1656284b68e86ef11ed8658866ac818a314" Jul 08 19:45:13.783531 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:45:13.783512 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b40fc666b7bcb1342d321107a011d1656284b68e86ef11ed8658866ac818a314\": container with ID starting with b40fc666b7bcb1342d321107a011d1656284b68e86ef11ed8658866ac818a314 not found: ID does not exist" containerID="b40fc666b7bcb1342d321107a011d1656284b68e86ef11ed8658866ac818a314" Jul 08 19:45:13.783629 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:13.783540 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b40fc666b7bcb1342d321107a011d1656284b68e86ef11ed8658866ac818a314"} err="failed to get container status \"b40fc666b7bcb1342d321107a011d1656284b68e86ef11ed8658866ac818a314\": rpc error: code = NotFound desc = could not find container \"b40fc666b7bcb1342d321107a011d1656284b68e86ef11ed8658866ac818a314\": container with ID starting with b40fc666b7bcb1342d321107a011d1656284b68e86ef11ed8658866ac818a314 not found: ID does not exist" Jul 08 19:45:13.878281 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:13.878244 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hfhh\" (UniqueName: \"kubernetes.io/projected/02739959-e191-436d-9c31-8388093e9501-kube-api-access-8hfhh\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:45:13.878281 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:13.878275 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02739959-e191-436d-9c31-8388093e9501-maas-api-tls\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:45:13.878281 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:13.878285 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02739959-e191-436d-9c31-8388093e9501-openshift-service-ca\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:45:14.095857 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:14.095825 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d86987-6755859574-z98zr"] Jul 08 19:45:14.100155 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:14.100131 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d86987-6755859574-z98zr"] Jul 08 19:45:15.282951 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:15.282919 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02739959-e191-436d-9c31-8388093e9501" path="/var/lib/kubelet/pods/02739959-e191-436d-9c31-8388093e9501/volumes" Jul 08 19:45:19.031221 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:19.031185 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:45:19.101037 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:19.101001 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:45:19.282983 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:19.282902 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:45:19.661296 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:19.661260 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:45:20.047425 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:20.047308 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:45:20.148728 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:20.148661 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:45:20.246487 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:20.246439 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:45:20.349498 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:20.349410 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4a9cb4-7bb87f5448-rngth"] Jul 08 19:45:20.349712 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:20.349683 2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4a9cb4-7bb87f5448-rngth" podUID="433dc1c4-8c47-4f00-bc80-518fff9ca429" containerName="maas-api" containerID="cri-o://b0f38d5c94eaa2617b10d608b535e4be4926f2859c46329508acb9434da2a2d4" gracePeriod=30 Jul 08 19:45:20.369703 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:20.369666 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4a9cb4-29725665-mpw8m"] Jul 08 19:45:20.369957 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:20.369919 2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4a9cb4-29725665-mpw8m" podUID="7840aff8-3d34-4255-978f-4d9d299be7e8" containerName="cleanup" containerID="cri-o://459ff971a025e3149e99964a23f44d6e17b0e98924bee977543b124b159e7758" gracePeriod=30 Jul 08 19:45:20.382105 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:20.382073 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:45:20.599240 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:20.599212 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4a9cb4-7bb87f5448-rngth" Jul 08 19:45:20.737903 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:20.737871 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/433dc1c4-8c47-4f00-bc80-518fff9ca429-maas-api-tls\") pod \"433dc1c4-8c47-4f00-bc80-518fff9ca429\" (UID: \"433dc1c4-8c47-4f00-bc80-518fff9ca429\") " Jul 08 19:45:20.738095 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:20.737909 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/433dc1c4-8c47-4f00-bc80-518fff9ca429-openshift-service-ca\") pod \"433dc1c4-8c47-4f00-bc80-518fff9ca429\" (UID: \"433dc1c4-8c47-4f00-bc80-518fff9ca429\") " Jul 08 19:45:20.738095 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:20.737952 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prz54\" (UniqueName: \"kubernetes.io/projected/433dc1c4-8c47-4f00-bc80-518fff9ca429-kube-api-access-prz54\") pod \"433dc1c4-8c47-4f00-bc80-518fff9ca429\" (UID: \"433dc1c4-8c47-4f00-bc80-518fff9ca429\") " Jul 08 19:45:20.738363 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:20.738334 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/433dc1c4-8c47-4f00-bc80-518fff9ca429-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "433dc1c4-8c47-4f00-bc80-518fff9ca429" (UID: "433dc1c4-8c47-4f00-bc80-518fff9ca429"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:45:20.740095 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:20.740070 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/433dc1c4-8c47-4f00-bc80-518fff9ca429-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "433dc1c4-8c47-4f00-bc80-518fff9ca429" (UID: "433dc1c4-8c47-4f00-bc80-518fff9ca429"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:45:20.740195 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:20.740071 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/433dc1c4-8c47-4f00-bc80-518fff9ca429-kube-api-access-prz54" (OuterVolumeSpecName: "kube-api-access-prz54") pod "433dc1c4-8c47-4f00-bc80-518fff9ca429" (UID: "433dc1c4-8c47-4f00-bc80-518fff9ca429"). InnerVolumeSpecName "kube-api-access-prz54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:45:20.816927 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:20.816891 2558 generic.go:356] "Generic (PLEG): container finished" podID="433dc1c4-8c47-4f00-bc80-518fff9ca429" containerID="b0f38d5c94eaa2617b10d608b535e4be4926f2859c46329508acb9434da2a2d4" exitCode=0 Jul 08 19:45:20.817128 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:20.816929 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4a9cb4-7bb87f5448-rngth" Jul 08 19:45:20.817128 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:20.816981 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4a9cb4-7bb87f5448-rngth" event={"ID":"433dc1c4-8c47-4f00-bc80-518fff9ca429","Type":"ContainerDied","Data":"b0f38d5c94eaa2617b10d608b535e4be4926f2859c46329508acb9434da2a2d4"} Jul 08 19:45:20.817128 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:20.817029 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4a9cb4-7bb87f5448-rngth" event={"ID":"433dc1c4-8c47-4f00-bc80-518fff9ca429","Type":"ContainerDied","Data":"34efc2d058ad50e6b66e24d5bfbece5c8ed195d46e322f18acc637855ae39233"} Jul 08 19:45:20.817128 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:20.817055 2558 scope.go:117] "RemoveContainer" containerID="b0f38d5c94eaa2617b10d608b535e4be4926f2859c46329508acb9434da2a2d4" Jul 08 19:45:20.826736 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:20.826719 2558 scope.go:117] "RemoveContainer" containerID="b0f38d5c94eaa2617b10d608b535e4be4926f2859c46329508acb9434da2a2d4" Jul 08 19:45:20.826965 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:45:20.826948 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0f38d5c94eaa2617b10d608b535e4be4926f2859c46329508acb9434da2a2d4\": container with ID starting with b0f38d5c94eaa2617b10d608b535e4be4926f2859c46329508acb9434da2a2d4 not found: ID does not exist" containerID="b0f38d5c94eaa2617b10d608b535e4be4926f2859c46329508acb9434da2a2d4" Jul 08 19:45:20.827015 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:20.826974 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0f38d5c94eaa2617b10d608b535e4be4926f2859c46329508acb9434da2a2d4"} err="failed to get container status \"b0f38d5c94eaa2617b10d608b535e4be4926f2859c46329508acb9434da2a2d4\": rpc error: code = NotFound desc = could not find container \"b0f38d5c94eaa2617b10d608b535e4be4926f2859c46329508acb9434da2a2d4\": container with ID starting with b0f38d5c94eaa2617b10d608b535e4be4926f2859c46329508acb9434da2a2d4 not found: ID does not exist" Jul 08 19:45:20.839488 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:20.839461 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prz54\" (UniqueName: \"kubernetes.io/projected/433dc1c4-8c47-4f00-bc80-518fff9ca429-kube-api-access-prz54\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:45:20.839573 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:20.839493 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/433dc1c4-8c47-4f00-bc80-518fff9ca429-maas-api-tls\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:45:20.839573 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:20.839509 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/433dc1c4-8c47-4f00-bc80-518fff9ca429-openshift-service-ca\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:45:20.840635 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:20.840615 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4a9cb4-7bb87f5448-rngth"] Jul 08 19:45:20.844658 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:20.844638 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4a9cb4-7bb87f5448-rngth"] Jul 08 19:45:21.281981 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:21.281948 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="433dc1c4-8c47-4f00-bc80-518fff9ca429" path="/var/lib/kubelet/pods/433dc1c4-8c47-4f00-bc80-518fff9ca429/volumes" Jul 08 19:45:25.972315 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:25.972282 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:45:26.072513 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:26.072472 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:45:26.230137 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:26.230045 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:45:31.696533 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:31.696447 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-796d55d59-7p887_07700279-0333-4aee-ab4b-2ac005a6ec83/manager/0.log" Jul 08 19:45:32.041474 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:32.041398 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-994cc9f69-2t2nf_02e501c0-f92d-4d99-946e-2d2344c1dd99/copy-manifests/0.log" Jul 08 19:45:32.059370 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:32.059338 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-994cc9f69-2t2nf_02e501c0-f92d-4d99-946e-2d2344c1dd99/manager/0.log" Jul 08 19:45:33.101471 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:33.101439 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wc2k8_bb55904d-f1a5-4a12-b403-d54a91df7de2/util/0.log" Jul 08 19:45:33.107091 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:33.107069 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wc2k8_bb55904d-f1a5-4a12-b403-d54a91df7de2/pull/0.log" Jul 08 19:45:33.112792 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:33.112756 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wc2k8_bb55904d-f1a5-4a12-b403-d54a91df7de2/extract/0.log" Jul 08 19:45:33.221144 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:33.221112 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mgrpx_bb40b1b2-9e96-46d1-9e8e-f0586f3938b7/util/0.log" Jul 08 19:45:33.227106 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:33.227081 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mgrpx_bb40b1b2-9e96-46d1-9e8e-f0586f3938b7/pull/0.log" Jul 08 19:45:33.232350 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:33.232328 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mgrpx_bb40b1b2-9e96-46d1-9e8e-f0586f3938b7/extract/0.log" Jul 08 19:45:33.346346 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:33.346317 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739vgr6_e758504b-8797-4466-9e69-b639a4694e6d/util/0.log" Jul 08 19:45:33.361435 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:33.361373 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739vgr6_e758504b-8797-4466-9e69-b639a4694e6d/pull/0.log" Jul 08 19:45:33.377395 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:33.377372 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739vgr6_e758504b-8797-4466-9e69-b639a4694e6d/extract/0.log" Jul 08 19:45:33.489315 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:33.489288 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lqtlg_c24b2a5d-6bf4-4d86-a669-bcb498b921be/extract/0.log" Jul 08 19:45:33.494542 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:33.494516 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lqtlg_c24b2a5d-6bf4-4d86-a669-bcb498b921be/util/0.log" Jul 08 19:45:33.500124 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:33.500100 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lqtlg_c24b2a5d-6bf4-4d86-a669-bcb498b921be/pull/0.log" Jul 08 19:45:34.910252 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:34.910220 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-s2rml_36f6816d-0f61-47ab-a861-fffd324ea5cf/discovery/0.log" Jul 08 19:45:35.372563 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:35.372512 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-vtjcn_434297f0-bb7f-4957-adcb-101410815ed8/payload-pre-processing/0.log" Jul 08 19:45:35.499019 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:35.498989 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-7cj9f_a41a36b0-53e5-4928-9147-03e60bf26b42/payload-processing/0.log" Jul 08 19:45:35.615170 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:35.615139 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7556c5577f-p4h6b_18b01b38-397e-492c-80c1-61217e74736f/router/0.log" Jul 08 19:45:35.962514 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:35.962488 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-l56tl_a8fd614e-0542-4c5c-997b-683bca306081/main/0.log" Jul 08 19:45:36.302351 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:36.302248 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4gc6w_c255c6e4-3338-4ded-bb5d-66022aa7d036/main/0.log" Jul 08 19:45:40.437649 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:40.437612 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:45:41.470942 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:41.470910 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" limit=1024 Jul 08 19:45:43.160121 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:43.160091 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mlw2c_640044e9-7c83-4f5a-9c9d-2e4183284806/global-pull-secret-syncer/0.log" Jul 08 19:45:43.304011 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:43.303984 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jwprt_17f1251c-a915-41c2-a44a-fd7a048826af/konnectivity-agent/0.log" Jul 08 19:45:43.402595 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:43.402556 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-191.ec2.internal_ce1e796271b5d048919b6c245c7db2fc/haproxy/0.log" Jul 08 19:45:43.404384 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:43.404363 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d86987-29725665-2srfn" Jul 08 19:45:43.528854 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:43.528783 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nt5hd\" (UniqueName: \"kubernetes.io/projected/891db0cb-c6a4-4cac-a0c7-6bca3a4711d6-kube-api-access-nt5hd\") pod \"891db0cb-c6a4-4cac-a0c7-6bca3a4711d6\" (UID: \"891db0cb-c6a4-4cac-a0c7-6bca3a4711d6\") " Jul 08 19:45:43.530774 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:43.530753 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/891db0cb-c6a4-4cac-a0c7-6bca3a4711d6-kube-api-access-nt5hd" (OuterVolumeSpecName: "kube-api-access-nt5hd") pod "891db0cb-c6a4-4cac-a0c7-6bca3a4711d6" (UID: "891db0cb-c6a4-4cac-a0c7-6bca3a4711d6"). InnerVolumeSpecName "kube-api-access-nt5hd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:45:43.629645 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:43.629607 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt5hd\" (UniqueName: \"kubernetes.io/projected/891db0cb-c6a4-4cac-a0c7-6bca3a4711d6-kube-api-access-nt5hd\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:45:43.946138 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:43.946103 2558 generic.go:356] "Generic (PLEG): container finished" podID="891db0cb-c6a4-4cac-a0c7-6bca3a4711d6" containerID="cf366d5d8237890f612bf50369d190ddc67311186c8aa36e6a6d2fb24f4b8301" exitCode=137 Jul 08 19:45:43.946342 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:43.946146 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d86987-29725665-2srfn" Jul 08 19:45:43.946342 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:43.946184 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d86987-29725665-2srfn" event={"ID":"891db0cb-c6a4-4cac-a0c7-6bca3a4711d6","Type":"ContainerDied","Data":"cf366d5d8237890f612bf50369d190ddc67311186c8aa36e6a6d2fb24f4b8301"} Jul 08 19:45:43.946342 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:43.946240 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d86987-29725665-2srfn" event={"ID":"891db0cb-c6a4-4cac-a0c7-6bca3a4711d6","Type":"ContainerDied","Data":"0f935a82acdfd00bc66da25965a471a381ed097deb7569a8cc7dd37c82331976"} Jul 08 19:45:43.946342 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:43.946262 2558 scope.go:117] "RemoveContainer" containerID="cf366d5d8237890f612bf50369d190ddc67311186c8aa36e6a6d2fb24f4b8301" Jul 08 19:45:43.954994 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:43.954976 2558 scope.go:117] "RemoveContainer" containerID="cf366d5d8237890f612bf50369d190ddc67311186c8aa36e6a6d2fb24f4b8301" Jul 08 19:45:43.955260 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:45:43.955240 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf366d5d8237890f612bf50369d190ddc67311186c8aa36e6a6d2fb24f4b8301\": container with ID starting with cf366d5d8237890f612bf50369d190ddc67311186c8aa36e6a6d2fb24f4b8301 not found: ID does not exist" containerID="cf366d5d8237890f612bf50369d190ddc67311186c8aa36e6a6d2fb24f4b8301" Jul 08 19:45:43.955308 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:43.955268 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf366d5d8237890f612bf50369d190ddc67311186c8aa36e6a6d2fb24f4b8301"} err="failed to get container status \"cf366d5d8237890f612bf50369d190ddc67311186c8aa36e6a6d2fb24f4b8301\": rpc error: code = NotFound desc = could not find container \"cf366d5d8237890f612bf50369d190ddc67311186c8aa36e6a6d2fb24f4b8301\": container with ID starting with cf366d5d8237890f612bf50369d190ddc67311186c8aa36e6a6d2fb24f4b8301 not found: ID does not exist" Jul 08 19:45:43.966859 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:43.966822 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d86987-29725665-2srfn"] Jul 08 19:45:43.973629 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:43.971683 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d86987-29725665-2srfn"] Jul 08 19:45:45.281294 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:45.281261 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="891db0cb-c6a4-4cac-a0c7-6bca3a4711d6" path="/var/lib/kubelet/pods/891db0cb-c6a4-4cac-a0c7-6bca3a4711d6/volumes" Jul 08 19:45:48.238821 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:48.238789 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wc2k8_bb55904d-f1a5-4a12-b403-d54a91df7de2/extract/0.log" Jul 08 19:45:48.264902 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:48.264876 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wc2k8_bb55904d-f1a5-4a12-b403-d54a91df7de2/util/0.log" Jul 08 19:45:48.294791 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:48.294766 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759wc2k8_bb55904d-f1a5-4a12-b403-d54a91df7de2/pull/0.log" Jul 08 19:45:48.324699 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:48.324675 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mgrpx_bb40b1b2-9e96-46d1-9e8e-f0586f3938b7/extract/0.log" Jul 08 19:45:48.350300 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:48.350278 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mgrpx_bb40b1b2-9e96-46d1-9e8e-f0586f3938b7/util/0.log" Jul 08 19:45:48.379628 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:48.379600 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mgrpx_bb40b1b2-9e96-46d1-9e8e-f0586f3938b7/pull/0.log" Jul 08 19:45:48.426727 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:48.426705 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739vgr6_e758504b-8797-4466-9e69-b639a4694e6d/extract/0.log" Jul 08 19:45:48.459861 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:48.459840 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739vgr6_e758504b-8797-4466-9e69-b639a4694e6d/util/0.log" Jul 08 19:45:48.494327 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:48.494265 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739vgr6_e758504b-8797-4466-9e69-b639a4694e6d/pull/0.log" Jul 08 19:45:48.533340 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:48.533314 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lqtlg_c24b2a5d-6bf4-4d86-a669-bcb498b921be/extract/0.log" Jul 08 19:45:48.564355 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:48.564326 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lqtlg_c24b2a5d-6bf4-4d86-a669-bcb498b921be/util/0.log" Jul 08 19:45:48.596804 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:48.596779 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lqtlg_c24b2a5d-6bf4-4d86-a669-bcb498b921be/pull/0.log" Jul 08 19:45:50.559763 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:50.559736 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4a9cb4-29725665-mpw8m" Jul 08 19:45:50.688855 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:50.688771 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrdkb\" (UniqueName: \"kubernetes.io/projected/7840aff8-3d34-4255-978f-4d9d299be7e8-kube-api-access-xrdkb\") pod \"7840aff8-3d34-4255-978f-4d9d299be7e8\" (UID: \"7840aff8-3d34-4255-978f-4d9d299be7e8\") " Jul 08 19:45:50.690889 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:50.690867 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7840aff8-3d34-4255-978f-4d9d299be7e8-kube-api-access-xrdkb" (OuterVolumeSpecName: "kube-api-access-xrdkb") pod "7840aff8-3d34-4255-978f-4d9d299be7e8" (UID: "7840aff8-3d34-4255-978f-4d9d299be7e8"). InnerVolumeSpecName "kube-api-access-xrdkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:45:50.790234 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:50.790197 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrdkb\" (UniqueName: \"kubernetes.io/projected/7840aff8-3d34-4255-978f-4d9d299be7e8-kube-api-access-xrdkb\") on node \"ip-10-0-131-191.ec2.internal\" DevicePath \"\"" Jul 08 19:45:50.983425 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:50.983329 2558 generic.go:356] "Generic (PLEG): container finished" podID="7840aff8-3d34-4255-978f-4d9d299be7e8" containerID="459ff971a025e3149e99964a23f44d6e17b0e98924bee977543b124b159e7758" exitCode=137 Jul 08 19:45:50.983425 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:50.983380 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4a9cb4-29725665-mpw8m" Jul 08 19:45:50.983425 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:50.983407 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4a9cb4-29725665-mpw8m" event={"ID":"7840aff8-3d34-4255-978f-4d9d299be7e8","Type":"ContainerDied","Data":"459ff971a025e3149e99964a23f44d6e17b0e98924bee977543b124b159e7758"} Jul 08 19:45:50.983755 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:50.983472 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4a9cb4-29725665-mpw8m" event={"ID":"7840aff8-3d34-4255-978f-4d9d299be7e8","Type":"ContainerDied","Data":"3168f0d9e29efeacf0afea2f48d58abc29affdb6ff29504fa49b686c8e42da43"} Jul 08 19:45:50.983755 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:50.983501 2558 scope.go:117] "RemoveContainer" containerID="459ff971a025e3149e99964a23f44d6e17b0e98924bee977543b124b159e7758" Jul 08 19:45:50.994429 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:50.994408 2558 scope.go:117] "RemoveContainer" containerID="459ff971a025e3149e99964a23f44d6e17b0e98924bee977543b124b159e7758" Jul 08 19:45:50.994757 ip-10-0-131-191 kubenswrapper[2558]: E0708 19:45:50.994736 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"459ff971a025e3149e99964a23f44d6e17b0e98924bee977543b124b159e7758\": container with ID starting with 459ff971a025e3149e99964a23f44d6e17b0e98924bee977543b124b159e7758 not found: ID does not exist" containerID="459ff971a025e3149e99964a23f44d6e17b0e98924bee977543b124b159e7758" Jul 08 19:45:50.994847 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:50.994765 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"459ff971a025e3149e99964a23f44d6e17b0e98924bee977543b124b159e7758"} err="failed to get container status \"459ff971a025e3149e99964a23f44d6e17b0e98924bee977543b124b159e7758\": rpc error: code = NotFound desc = could not find container \"459ff971a025e3149e99964a23f44d6e17b0e98924bee977543b124b159e7758\": container with ID starting with 459ff971a025e3149e99964a23f44d6e17b0e98924bee977543b124b159e7758 not found: ID does not exist" Jul 08 19:45:51.006312 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:51.006281 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4a9cb4-29725665-mpw8m"] Jul 08 19:45:51.012087 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:51.012064 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4a9cb4-29725665-mpw8m"] Jul 08 19:45:51.282062 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:51.281985 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7840aff8-3d34-4255-978f-4d9d299be7e8" path="/var/lib/kubelet/pods/7840aff8-3d34-4255-978f-4d9d299be7e8/volumes" Jul 08 19:45:51.408616 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:51.408568 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-r7czv_9cebd65f-6a73-46e8-9995-7797c67acb53/cluster-monitoring-operator/0.log" Jul 08 19:45:51.709334 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:51.709298 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z4mqp_e4f979e3-704b-4b38-b0fd-765ae6958073/node-exporter/0.log" Jul 08 19:45:51.728470 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:51.728446 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z4mqp_e4f979e3-704b-4b38-b0fd-765ae6958073/kube-rbac-proxy/0.log" Jul 08 19:45:51.748191 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:51.748170 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z4mqp_e4f979e3-704b-4b38-b0fd-765ae6958073/init-textfile/0.log" Jul 08 19:45:52.064619 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:52.064511 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-7ssjd_3aa66b57-e57d-4411-8e4a-4afa9e33b34b/prometheus-operator-admission-webhook/0.log" Jul 08 19:45:53.088137 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:53.088101 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fqj4d/perf-node-gather-daemonset-qvd8v"] Jul 08 19:45:53.088627 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:53.088566 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02739959-e191-436d-9c31-8388093e9501" containerName="maas-api" Jul 08 19:45:53.088627 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:53.088596 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="02739959-e191-436d-9c31-8388093e9501" containerName="maas-api" Jul 08 19:45:53.088627 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:53.088617 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7840aff8-3d34-4255-978f-4d9d299be7e8" containerName="cleanup" Jul 08 19:45:53.088627 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:53.088625 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7840aff8-3d34-4255-978f-4d9d299be7e8" containerName="cleanup" Jul 08 19:45:53.088769 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:53.088636 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="891db0cb-c6a4-4cac-a0c7-6bca3a4711d6" containerName="cleanup" Jul 08 19:45:53.088769 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:53.088644 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="891db0cb-c6a4-4cac-a0c7-6bca3a4711d6" containerName="cleanup" Jul 08 19:45:53.088769 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:53.088654 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="433dc1c4-8c47-4f00-bc80-518fff9ca429" containerName="maas-api" Jul 08 19:45:53.088769 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:53.088659 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="433dc1c4-8c47-4f00-bc80-518fff9ca429" containerName="maas-api" Jul 08 19:45:53.088769 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:53.088726 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="02739959-e191-436d-9c31-8388093e9501" containerName="maas-api" Jul 08 19:45:53.088769 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:53.088735 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="891db0cb-c6a4-4cac-a0c7-6bca3a4711d6" containerName="cleanup" Jul 08 19:45:53.088769 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:53.088742 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="433dc1c4-8c47-4f00-bc80-518fff9ca429" containerName="maas-api" Jul 08 19:45:53.088769 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:53.088748 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7840aff8-3d34-4255-978f-4d9d299be7e8" containerName="cleanup" Jul 08 19:45:53.093024 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:53.093009 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fqj4d/perf-node-gather-daemonset-qvd8v" Jul 08 19:45:53.095479 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:53.095458 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fqj4d\"/\"openshift-service-ca.crt\"" Jul 08 19:45:53.096352 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:53.096334 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fqj4d\"/\"kube-root-ca.crt\"" Jul 08 19:45:53.096399 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:53.096334 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fqj4d\"/\"default-dockercfg-m444m\"" Jul 08 19:45:53.100783 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:53.100763 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fqj4d/perf-node-gather-daemonset-qvd8v"] Jul 08 19:45:53.211742 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:53.211706 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e6cc192b-d624-4b33-b17e-fb227c4de6e7-proc\") pod \"perf-node-gather-daemonset-qvd8v\" (UID: \"e6cc192b-d624-4b33-b17e-fb227c4de6e7\") " pod="openshift-must-gather-fqj4d/perf-node-gather-daemonset-qvd8v" Jul 08 19:45:53.211742 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:53.211748 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e6cc192b-d624-4b33-b17e-fb227c4de6e7-lib-modules\") pod \"perf-node-gather-daemonset-qvd8v\" (UID: \"e6cc192b-d624-4b33-b17e-fb227c4de6e7\") " pod="openshift-must-gather-fqj4d/perf-node-gather-daemonset-qvd8v" Jul 08 19:45:53.211941 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:53.211827 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6cc192b-d624-4b33-b17e-fb227c4de6e7-sys\") pod \"perf-node-gather-daemonset-qvd8v\" (UID: \"e6cc192b-d624-4b33-b17e-fb227c4de6e7\") " pod="openshift-must-gather-fqj4d/perf-node-gather-daemonset-qvd8v" Jul 08 19:45:53.211941 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:53.211886 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e6cc192b-d624-4b33-b17e-fb227c4de6e7-podres\") pod \"perf-node-gather-daemonset-qvd8v\" (UID: \"e6cc192b-d624-4b33-b17e-fb227c4de6e7\") " pod="openshift-must-gather-fqj4d/perf-node-gather-daemonset-qvd8v" Jul 08 19:45:53.211941 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:53.211930 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w74x7\" (UniqueName: \"kubernetes.io/projected/e6cc192b-d624-4b33-b17e-fb227c4de6e7-kube-api-access-w74x7\") pod \"perf-node-gather-daemonset-qvd8v\" (UID: \"e6cc192b-d624-4b33-b17e-fb227c4de6e7\") " pod="openshift-must-gather-fqj4d/perf-node-gather-daemonset-qvd8v" Jul 08 19:45:53.313131 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:53.313098 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e6cc192b-d624-4b33-b17e-fb227c4de6e7-proc\") pod \"perf-node-gather-daemonset-qvd8v\" (UID: \"e6cc192b-d624-4b33-b17e-fb227c4de6e7\") " pod="openshift-must-gather-fqj4d/perf-node-gather-daemonset-qvd8v" Jul 08 19:45:53.313131 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:53.313136 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e6cc192b-d624-4b33-b17e-fb227c4de6e7-lib-modules\") pod \"perf-node-gather-daemonset-qvd8v\" (UID: \"e6cc192b-d624-4b33-b17e-fb227c4de6e7\") " pod="openshift-must-gather-fqj4d/perf-node-gather-daemonset-qvd8v" Jul 08 19:45:53.313364 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:53.313189 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6cc192b-d624-4b33-b17e-fb227c4de6e7-sys\") pod \"perf-node-gather-daemonset-qvd8v\" (UID: \"e6cc192b-d624-4b33-b17e-fb227c4de6e7\") " pod="openshift-must-gather-fqj4d/perf-node-gather-daemonset-qvd8v" Jul 08 19:45:53.313364 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:53.313214 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e6cc192b-d624-4b33-b17e-fb227c4de6e7-podres\") pod \"perf-node-gather-daemonset-qvd8v\" (UID: \"e6cc192b-d624-4b33-b17e-fb227c4de6e7\") " pod="openshift-must-gather-fqj4d/perf-node-gather-daemonset-qvd8v" Jul 08 19:45:53.313364 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:53.313247 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w74x7\" (UniqueName: \"kubernetes.io/projected/e6cc192b-d624-4b33-b17e-fb227c4de6e7-kube-api-access-w74x7\") pod \"perf-node-gather-daemonset-qvd8v\" (UID: \"e6cc192b-d624-4b33-b17e-fb227c4de6e7\") " pod="openshift-must-gather-fqj4d/perf-node-gather-daemonset-qvd8v" Jul 08 19:45:53.313364 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:53.313307 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6cc192b-d624-4b33-b17e-fb227c4de6e7-sys\") pod \"perf-node-gather-daemonset-qvd8v\" (UID: \"e6cc192b-d624-4b33-b17e-fb227c4de6e7\") " pod="openshift-must-gather-fqj4d/perf-node-gather-daemonset-qvd8v" Jul 08 19:45:53.313364 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:53.313311 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e6cc192b-d624-4b33-b17e-fb227c4de6e7-lib-modules\") pod \"perf-node-gather-daemonset-qvd8v\" (UID: \"e6cc192b-d624-4b33-b17e-fb227c4de6e7\") " pod="openshift-must-gather-fqj4d/perf-node-gather-daemonset-qvd8v" Jul 08 19:45:53.313364 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:53.313330 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e6cc192b-d624-4b33-b17e-fb227c4de6e7-proc\") pod \"perf-node-gather-daemonset-qvd8v\" (UID: \"e6cc192b-d624-4b33-b17e-fb227c4de6e7\") " pod="openshift-must-gather-fqj4d/perf-node-gather-daemonset-qvd8v" Jul 08 19:45:53.313364 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:53.313358 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e6cc192b-d624-4b33-b17e-fb227c4de6e7-podres\") pod \"perf-node-gather-daemonset-qvd8v\" (UID: \"e6cc192b-d624-4b33-b17e-fb227c4de6e7\") " pod="openshift-must-gather-fqj4d/perf-node-gather-daemonset-qvd8v" Jul 08 19:45:53.321570 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:53.321549 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w74x7\" (UniqueName: \"kubernetes.io/projected/e6cc192b-d624-4b33-b17e-fb227c4de6e7-kube-api-access-w74x7\") pod \"perf-node-gather-daemonset-qvd8v\" (UID: \"e6cc192b-d624-4b33-b17e-fb227c4de6e7\") " pod="openshift-must-gather-fqj4d/perf-node-gather-daemonset-qvd8v" Jul 08 19:45:53.403716 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:53.403682 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fqj4d/perf-node-gather-daemonset-qvd8v" Jul 08 19:45:53.412595 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:53.412552 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-pr89r_a06e8e9a-6466-46ce-acaa-abdda774607d/networking-console-plugin/0.log" Jul 08 19:45:53.536706 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:53.536682 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fqj4d/perf-node-gather-daemonset-qvd8v"] Jul 08 19:45:53.538316 ip-10-0-131-191 kubenswrapper[2558]: W0708 19:45:53.538280 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode6cc192b_d624_4b33_b17e_fb227c4de6e7.slice/crio-ed2da36e8fa2bfe258b98dd4618ed12bbf389b3846b25d2f07bb90ca3a1f39d5 WatchSource:0}: Error finding container ed2da36e8fa2bfe258b98dd4618ed12bbf389b3846b25d2f07bb90ca3a1f39d5: Status 404 returned error can't find the container with id ed2da36e8fa2bfe258b98dd4618ed12bbf389b3846b25d2f07bb90ca3a1f39d5 Jul 08 19:45:54.000890 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:54.000851 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fqj4d/perf-node-gather-daemonset-qvd8v" event={"ID":"e6cc192b-d624-4b33-b17e-fb227c4de6e7","Type":"ContainerStarted","Data":"8774d63508988dc9db7bef3c1ba8e63c52b8e925c3ccd54096fa6d14dc8c0dca"} Jul 08 19:45:54.001077 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:54.000914 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fqj4d/perf-node-gather-daemonset-qvd8v" Jul 08 19:45:54.001077 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:54.000938 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fqj4d/perf-node-gather-daemonset-qvd8v" event={"ID":"e6cc192b-d624-4b33-b17e-fb227c4de6e7","Type":"ContainerStarted","Data":"ed2da36e8fa2bfe258b98dd4618ed12bbf389b3846b25d2f07bb90ca3a1f39d5"} Jul 08 19:45:54.051850 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:54.051799 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fqj4d/perf-node-gather-daemonset-qvd8v" podStartSLOduration=1.051784765 podStartE2EDuration="1.051784765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:45:54.049922916 +0000 UTC m=+2537.308875840" watchObservedRunningTime="2026-07-08 19:45:54.051784765 +0000 UTC m=+2537.310737690" Jul 08 19:45:54.108317 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:54.108287 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-54hnr_1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a/console-operator/2.log" Jul 08 19:45:54.116398 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:54.116375 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-54hnr_1ebcfaa0-2d87-4a85-9e2e-13d4e230e39a/console-operator/3.log" Jul 08 19:45:55.154631 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:55.154597 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-69gjd_3fd49589-6210-4804-bccd-079d96770654/volume-data-source-validator/0.log" Jul 08 19:45:56.210562 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:56.210535 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ndtk5_eae871c8-0143-47c6-853c-2413d8f6fa70/dns/0.log" Jul 08 19:45:56.231127 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:56.231101 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ndtk5_eae871c8-0143-47c6-853c-2413d8f6fa70/kube-rbac-proxy/0.log" Jul 08 19:45:56.291508 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:56.291480 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-44598_74f5d0c6-337e-407c-bdef-75c0dcd3c47d/dns-node-resolver/0.log" Jul 08 19:45:56.917618 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:56.917570 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gktll_a0658e5c-4e22-43d3-ac11-86df5ff39969/node-ca/0.log" Jul 08 19:45:58.055919 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:58.055874 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-s2rml_36f6816d-0f61-47ab-a861-fffd324ea5cf/discovery/0.log" Jul 08 19:45:58.232484 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:58.232451 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-vtjcn_434297f0-bb7f-4957-adcb-101410815ed8/payload-pre-processing/0.log" Jul 08 19:45:58.308539 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:58.308439 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-7cj9f_a41a36b0-53e5-4928-9147-03e60bf26b42/payload-processing/0.log" Jul 08 19:45:58.334799 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:58.334769 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7556c5577f-p4h6b_18b01b38-397e-492c-80c1-61217e74736f/router/0.log" Jul 08 19:45:59.056174 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:59.056144 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lng6h_2e731fad-40cc-4fa8-92cf-5465ff9a29dd/serve-healthcheck-canary/0.log" Jul 08 19:45:59.553654 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:59.553621 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-l4fvs_9f2932e3-5510-44d8-bd92-76ebdac8ce3f/insights-operator/0.log" Jul 08 19:45:59.557064 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:59.557040 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-l4fvs_9f2932e3-5510-44d8-bd92-76ebdac8ce3f/insights-operator/1.log" Jul 08 19:45:59.709852 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:59.709820 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xd9ln_fd304abd-e907-409f-9faf-94c4b78be9d1/kube-rbac-proxy/0.log" Jul 08 19:45:59.728480 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:59.728459 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xd9ln_fd304abd-e907-409f-9faf-94c4b78be9d1/exporter/0.log" Jul 08 19:45:59.749064 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:45:59.749045 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xd9ln_fd304abd-e907-409f-9faf-94c4b78be9d1/extractor/0.log" Jul 08 19:46:00.010508 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:46:00.010474 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725665-z6bgf"] Jul 08 19:46:00.010710 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:46:00.010688 2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725665-z6bgf" podUID="b2bbf45f-e9d9-4548-83b4-07280d39dc05" containerName="cleanup" containerID="cri-o://05373a9607672edc87cdd6fcf169da1b0dd535a5453acdb867e42c6ef4515832" gracePeriod=30 Jul 08 19:46:00.016479 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:46:00.016452 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fqj4d/perf-node-gather-daemonset-qvd8v" Jul 08 19:46:02.319670 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:46:02.319637 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-796d55d59-7p887_07700279-0333-4aee-ab4b-2ac005a6ec83/manager/0.log" Jul 08 19:46:02.534999 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:46:02.534910 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-994cc9f69-2t2nf_02e501c0-f92d-4d99-946e-2d2344c1dd99/manager/0.log" Jul 08 19:46:02.551334 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:46:02.551313 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-994cc9f69-2t2nf_02e501c0-f92d-4d99-946e-2d2344c1dd99/copy-manifests/0.log" Jul 08 19:46:03.914713 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:46:03.914685 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-69dfbfd578-2h524_6f1b3c05-4c06-4f57-a739-20f77855c0ee/manager/0.log" Jul 08 19:46:04.321800 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:46:04.321767 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-s2rml_36f6816d-0f61-47ab-a861-fffd324ea5cf/discovery/0.log" Jul 08 19:46:04.475170 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:46:04.475131 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-vtjcn_434297f0-bb7f-4957-adcb-101410815ed8/payload-pre-processing/0.log" Jul 08 19:46:04.538947 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:46:04.538918 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-7cj9f_a41a36b0-53e5-4928-9147-03e60bf26b42/payload-processing/0.log" Jul 08 19:46:04.552636 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:46:04.552603 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7556c5577f-p4h6b_18b01b38-397e-492c-80c1-61217e74736f/router/0.log" Jul 08 19:46:05.875683 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:46:05.875640 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-s2rml_36f6816d-0f61-47ab-a861-fffd324ea5cf/discovery/0.log" Jul 08 19:46:06.014951 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:46:06.014921 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-vtjcn_434297f0-bb7f-4957-adcb-101410815ed8/payload-pre-processing/0.log" Jul 08 19:46:06.086331 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:46:06.086302 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-7cj9f_a41a36b0-53e5-4928-9147-03e60bf26b42/payload-processing/0.log" Jul 08 19:46:06.105774 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:46:06.105742 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7556c5577f-p4h6b_18b01b38-397e-492c-80c1-61217e74736f/router/0.log" Jul 08 19:46:09.297537 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:46:09.297452 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-7qzsf_58ce9cbf-4ed4-4482-8786-d92dd2f9a0e0/kube-storage-version-migrator-operator/1.log" Jul 08 19:46:09.299101 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:46:09.299080 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-7qzsf_58ce9cbf-4ed4-4482-8786-d92dd2f9a0e0/kube-storage-version-migrator-operator/0.log" Jul 08 19:46:10.325525 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:46:10.325491 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7ks2q_60145409-2aa4-4d68-9405-30b21fae674e/kube-multus-additional-cni-plugins/0.log" Jul 08 19:46:10.344744 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:46:10.344721 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7ks2q_60145409-2aa4-4d68-9405-30b21fae674e/egress-router-binary-copy/0.log" Jul 08 19:46:10.365726 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:46:10.365701 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7ks2q_60145409-2aa4-4d68-9405-30b21fae674e/cni-plugins/0.log" Jul 08 19:46:10.386446 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:46:10.386420 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7ks2q_60145409-2aa4-4d68-9405-30b21fae674e/bond-cni-plugin/0.log" Jul 08 19:46:10.407025 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:46:10.406997 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7ks2q_60145409-2aa4-4d68-9405-30b21fae674e/routeoverride-cni/0.log" Jul 08 19:46:10.427690 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:46:10.427667 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7ks2q_60145409-2aa4-4d68-9405-30b21fae674e/whereabouts-cni-bincopy/0.log" Jul 08 19:46:10.447245 ip-10-0-131-191 kubenswrapper[2558]: I0708 19:46:10.447224 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7ks2q_60145409-2aa4-4d68-9405-30b21fae674e/whereabouts-cni/0.log"